CN113434296A - Cache back-brushing method and device, electronic equipment and storage medium - Google Patents
Cache back-brushing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113434296A CN113434296A CN202110742870.6A CN202110742870A CN113434296A CN 113434296 A CN113434296 A CN 113434296A CN 202110742870 A CN202110742870 A CN 202110742870A CN 113434296 A CN113434296 A CN 113434296A
- Authority
- CN
- China
- Prior art keywords
- cache
- log
- brushing
- data block
- rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5022—Mechanisms to release resources
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides a cache refresh method, a cache refresh device, electronic equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: determining resource pressure, wherein the resource pressure comprises log disk usage; determining a cache refresh rate based on the resource pressure; and caching and back-brushing the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block based on the caching and back-brushing rate, wherein M and N are positive integers. According to the method, resource recovery and resource pressure are correlated, the rear-end I/O aggregation rate is adjusted according to the resource pressure, the throughput requirement when the resource pressure is high and the I/O aggregation efficiency when the resource pressure is low can be guaranteed, and meanwhile, the log resource recovery is effectively triggered by each time of back-flushing according to the data block sequence related to the log, so that the recovery efficiency of the log resource is stabilized.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a cache refresh method and apparatus, an electronic device, and a storage medium.
Background
In the existing log recovery and corresponding cache back-flushing technology, when an object is first dirtied, a cache is put into a back-flushing queue, a back-end thread back-flushing is waited, and an object dirtying timestamp is recorded; the back-brushing thread periodically checks the buffer pressure and the back-brushing queue, the back-brushing is triggered when the buffer pressure is larger or the dirty time of an object exceeds a certain time threshold, and the object with the dirty time exceeding the threshold is back-brushed according to the time stamp sequence; and after the back-flushing is finished, notifying the upper layer to clean the log and the associated resources.
Based on the above prior art conditions, under the condition that the resource pressure of the log disk is large, the constant back-brushing rate may cause the problem of current limitation when the log is not recovered timely due to the fact that the back-brushing is not timely.
In addition, the log module and the cache module are used as two independent modules, and the processes are closed respectively. But the log recovery is based on the buffer data to be back-brushed, the buffer module can not see the log, and the back-brushing strategy is independent. When the data corresponding to the log to be recovered is not refreshed timely, the log space cannot be recovered timely, the I/O of the upper-layer service is limited, and the frequent current limiting of the log under high service pressure and the jitter of the processing capacity of the system are caused.
Therefore, the existing cache back-flushing technology has the problem that the current is limited when the log recovery is not timely.
Disclosure of Invention
In view of the above, embodiments of the present disclosure provide a cache refresh method, an apparatus, an electronic device, and a storage medium, so as to solve the problem in the prior art that the current is limited when the log is not timely recovered in the existing cache refresh technology.
The embodiment of the application provides a cache back-brushing method, which comprises the following steps: determining resource pressure, wherein the resource pressure comprises log disk usage; determining a cache refresh rate based on the resource pressure; and caching and back-brushing the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block based on the caching and back-brushing rate, wherein M and N are positive integers.
In the implementation mode, the resource recovery is associated with the resource pressure, the rear-end I/O aggregation rate is adjusted according to the resource pressure, the throughput requirement when the resource pressure is high and the I/O aggregation efficiency when the resource pressure is low can be guaranteed, and meanwhile, the log resource recovery is effectively triggered by each back-flushing according to the sequence of the data blocks associated with the logs, so that the recovery efficiency of the log resources is stabilized.
Optionally, the determining resource pressure comprises: when the utilization rate of the log disk is less than or equal to a first preset utilization rate, determining that the resource pressure is sufficient; when the utilization rate of the log disk is greater than the first preset utilization rate and less than or equal to a second preset utilization rate, determining that the resource pressure is normal; and when the utilization rate of the log disk is greater than the second preset utilization rate, determining that the resource pressure is resource tension, wherein the first preset utilization rate is less than the second preset utilization rate.
In the implementation mode, the resource pressure is judged based on the utilization rate of the log disk, so that the subsequent cache back-flushing rate is adjusted, and the influence of the cache back-flushing on the limited resources recovered by the log is avoided.
Optionally, the determining a cache refresh rate based on the resource pressure includes: when the resources are sufficient, determining the cache back-brushing rate as quantitative back-brushing of the data blocks in the cache data; when the resources are normal, determining the cache back-brushing rate as performing cache back-brushing on the data blocks in the cache data based on a dirty time threshold; and when resources are in shortage, determining the cache back-brushing rate as the dirty time is not checked, and performing cache back-brushing on the data blocks in the cache data based on the sorting of the back-brushing queue.
In the above implementation, the brush is quantitatively brushed back when the resources are sufficient, and polymerization is preferentially carried out; when the resources are normal, caching and refreshing the data with overtime dirty time based on a dirty time threshold value, and ensuring the balance of aggregation and resource recovery; when resources are in shortage, the dirty setting time is not checked, the cache refresh rate is increased, and the resource recovery is accelerated; therefore, the adaptive cache back-brushing mode is respectively provided under different resource pressures, and the cache back-brushing rate can be converted into the log resource recovery efficiency equivalently.
Optionally, the performing quantitative back-brushing on the data block in the cache data includes: and performing cache back-flushing on a preset number of data blocks in the cache data every time a preset interval time passes.
In the implementation mode, when the resource pressure is low, the buffer memory is refreshed at fixed intervals and fixed quantity, the aggregation can be carried out to the maximum extent, and the aggregation efficiency is improved.
Optionally, the performing cache back-brushing on the data block in the cache data based on the dirty time threshold includes: determining dirty time according to the current time and the dirty time stamp of the data block in the cache data; and performing cache back-brushing on the data blocks in the cache data, wherein the dirty time of the data blocks exceeds the dirty time threshold.
In the implementation mode, the normal back-brushing rate is kept based on the dirty time when the resource pressure is normal, so that the balance of aggregation and resource recovery is ensured.
Optionally, before the flushing the specified data block associated with the specified log in the back-flushing queue based on the cache back-flushing rate and the previous data block and the next data block of the specified data block, the method further includes: when the designated data block associated with the designated log is dirty, putting the designated data block into the back-flushing queue according to the log order of the designated log; recording a dirty timestamp that the specified data block is dirty.
In the implementation manner, when the designated data block is dirty, the designated data block is put into the back-flushing queue according to the log sequence, so that the back-flushing queue for subsequently sorting the data blocks based on the log sequence determines the databases adjacent to the designated log based on the data block sorting and performs cache back-flushing, and each back-flushing can effectively promote recovery of released log space and related resources.
Optionally, after the flushing the specified data block associated with the specified log in the back-flushing queue based on the cache back-flushing rate and the previous data block and the next data block of the specified data block, the method further includes: and informing the upper layer equipment to clean the designated log in the log disk and the associated resources of the designated log.
In the implementation manner, after the designated data block associated with the designated log and the previous data block and the next data block are cached and flushed, the upper-layer device is notified to clean the designated log in the log disk, so that the successful recovery of the designated log can be ensured, and the stable conversion between the cache flushing rate and the resource recovery is ensured.
The embodiment of the present application further provides a cache brushing device, the device includes: the resource pressure determining module is used for determining resource pressure, and the resource pressure comprises log disk utilization rate; the back-brushing rate determining module is used for determining the cache back-brushing rate based on the resource pressure; and the cache back-brushing module is used for performing cache back-brushing on the specified data block associated with the specified log in the back-brushing queue, the front M data block and the rear N data block of the specified data block based on the cache back-brushing rate, wherein M and N are positive integers.
In the implementation mode, the resource recovery is associated with the resource pressure, the rear-end I/O aggregation rate is adjusted according to the resource pressure, the throughput requirement when the resource pressure is high and the I/O aggregation efficiency when the resource pressure is low can be guaranteed, and meanwhile, the log resource recovery is effectively triggered by each back-flushing according to the sequence of the data blocks associated with the logs, so that the recovery efficiency of the log resources is stabilized.
Optionally, the resource pressure determining module is specifically configured to: when the utilization rate of the log disk is less than or equal to a first preset utilization rate, determining that the resource pressure is sufficient; when the utilization rate of the log disk is greater than the first preset utilization rate and less than or equal to a second preset utilization rate, determining that the resource pressure is normal; and when the utilization rate of the log disk is greater than the second preset utilization rate, determining that the resource pressure is resource tension, wherein the first preset utilization rate is less than the second preset utilization rate.
In the implementation mode, the resource pressure is judged based on the utilization rate of the log disk, so that the subsequent cache back-flushing rate is adjusted, and the influence of the cache back-flushing on the limited resources recovered by the log is avoided.
Optionally, the refresh rate determination module is specifically configured to: when the resources are sufficient, determining the cache back-brushing rate as quantitative back-brushing of the data blocks in the cache data; when the resources are normal, determining the cache back-brushing rate as performing cache back-brushing on the data blocks in the cache data based on a dirty time threshold; and when resources are in shortage, determining the cache back-brushing rate as the dirty time is not checked, and performing cache back-brushing on the data blocks in the cache data based on the sorting of the back-brushing queue.
In the above implementation, the brush is quantitatively brushed back when the resources are sufficient, and polymerization is preferentially carried out; when the resources are normal, caching and refreshing the data with overtime dirty time based on a dirty time threshold value, and ensuring the balance of aggregation and resource recovery; when resources are in shortage, the dirty setting time is not checked, the cache refresh rate is increased, and the resource recovery is accelerated; therefore, the adaptive cache back-brushing mode is respectively provided under different resource pressures, and the cache back-brushing rate can be converted into the log resource recovery efficiency equivalently.
Optionally, the cache refresh module is specifically configured to: and performing cache back-flushing on a preset number of data blocks in the cache data every time a preset interval time passes.
In the implementation mode, when the resource pressure is low, the buffer memory is refreshed at fixed intervals and fixed quantity, the aggregation can be carried out to the maximum extent, and the aggregation efficiency is improved.
Optionally, the cache refresh module is specifically configured to: determining dirty time according to the current time and the dirty time stamp of the data block in the cache data; and performing cache back-brushing on the data blocks in the cache data, wherein the dirty time of the data blocks exceeds the dirty time threshold.
In the implementation mode, the normal back-brushing rate is kept based on the dirty time when the resource pressure is normal, so that the balance of aggregation and resource recovery is ensured.
Optionally, the cache refresh apparatus further includes: the dirty setting module is used for putting the specified data blocks into the back-brushing queue according to the log sequence of the specified log when the specified data blocks associated with the specified log are dirty; recording a dirty timestamp that the specified data block is dirty.
In the implementation manner, when the designated data block is dirty, the designated data block is put into the back-flushing queue according to the log sequence, so that the back-flushing queue for subsequently sorting the data blocks based on the log sequence determines the databases adjacent to the designated log based on the data block sorting and performs cache back-flushing, and each back-flushing can effectively promote recovery of released log space and related resources.
Optionally, the cache refresh apparatus further includes: and the notification module is used for notifying the upper-layer equipment to clean the designated log in the log disk and the associated resources of the designated log.
In the implementation manner, after the designated data block associated with the designated log and the previous data block and the next data block are cached and flushed, the upper-layer device is notified to clean the designated log in the log disk, so that the successful recovery of the designated log can be ensured, and the stable conversion between the cache flushing rate and the resource recovery is ensured.
An embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores program instructions, and the processor executes steps in any one of the above implementation manners when reading and executing the program instructions.
The embodiment of the present application further provides a readable storage medium, in which computer program instructions are stored, and the computer program instructions are read by a processor and executed to perform the steps in any of the above implementation manners.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a cache refresh method according to an embodiment of the present disclosure.
Fig. 2 is a schematic flowchart of a sequence step of a refresh queue according to an embodiment of the present disclosure.
Fig. 3 is a schematic block diagram of a cache refresh apparatus according to an embodiment of the present disclosure.
Icon: 20-cache refresh means; 21-a resource pressure determination module; 22-a brush-back rate determination module; 23-cache back-flush module.
Detailed Description
The technical solution in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
The research of the applicant finds that in the process of carrying out log recovery on the resource space of the log disk and carrying out cache refreshing, the log module mainly provides high availability for the metadata, so that the metadata node can not lose data due to the fact that the data is not written into the back end under the conditions of power failure, equipment failure and the like. The master log node is responsible for receiving logs submitted by the business layer, aggregating submitted log data, leading the curing operation of the logs, applying the log data to cache and the like. When the consistency of all log nodes is kept, the logs are solidified and written into a local log disk, the space of the log disk is limited, and invalid logs need to be timely recycled to release the log space under the condition that new logs need to be written repeatedly.
In addition, the cache module is used for accelerating metadata data access and I/O aggregation in the process so as to reduce access to a back-end disk. The cache module caches recently accessed data according to pages by using a data access locality principle. The cache module provides a read-write data access interface for metadata, the metadata reads data through cache, the cache hits and directly returns to a request page, and the cache misses and then reads and returns to an upper layer through a rear end, so that the metadata writes and directly writes into the cache and returns, meanwhile, a corresponding page is dirty, and then the corresponding page is cached and refreshed.
Wherein dirty indicates that the corresponding data block has been modified, and the marked data block needs to be flushed back to disk.
In the whole process of cache back-flushing and log recovery, the log module performs I/O aggregation on the log data generated by the metadata operation, and identifies the log order, for example, a unique and incremental log ID is used as the log order and written into a local log disk and other log copies, and after the log is applied to the cache, the log module is waited to back-flush the data and is notified that the log module can perform log recovery. Due to the fact that the metadata operation has a time sequence, log recovery also needs to keep the log sequence, and logs with continuously increasing log IDs are recovered in batches when the logs are recovered, and the space of a log disk where the logs are located is released.
Because the log module and the cache module are two independent modules, the processes are closed respectively, and meanwhile, log recovery depends on cache data to be flushed, recovery of corresponding logs can be performed only after data cache flushing corresponding to the logs is completed, the data flushing corresponding to the logs to be recovered is not timely, so that log space cannot be recovered timely, specifically, the cache flushing sequence and the log recovery sequence are not consistent, when the metadata pressure is higher, a large number of logs can be generated, cache flushing can be performed all the time, but the data cached and flushed cannot trigger corresponding log recovery, for example, the logs with the log ID of n are recovered, when the data flushed all the time after writing the data block corresponding to the log with the log ID of n +2, even if the corresponding logs are in a recoverable state, the logs with the log ID of n +1 cannot be recovered all the time, so that all the following logs are not recovered, the method causes log current limitation, and causes frequent current limitation of logs under high service pressure and jitter of system processing capacity. When the resource pressure is small, the cache refresh rate is high, which affects the back-end I/O aggregation efficiency.
In order to solve the above problem, an embodiment of the present application provides a cache refresh method, please refer to fig. 1, where fig. 1 is a schematic flow chart of the cache refresh method provided in the embodiment of the present application, and the specific steps of the cache refresh method may be as follows:
step S12: determining resource pressure, wherein the resource pressure comprises log disk usage.
It should be understood that the resource pressure can be customized by the upper layer user, and is generally divided according to the usage proportion of the limited resources relying on cache refresh for resource recovery, such as the service layer memory, the log disk usage rate, and the like.
In this embodiment, taking the usage rate of the log disk as an example, the front-end I/O may occupy the space of the log disk when writing log data, and the recovery of the log on the log disk may be performed only after the corresponding cache data is cached and refreshed, and a user may set that the resource pressure is sufficient when the usage rate of the log disk is less than or equal to a first preset usage rate, the resource pressure is normal when the usage rate of the log disk is greater than the first preset usage rate and less than or equal to a second preset usage rate, and the resource pressure is resource shortage when the usage rate of the log disk is greater than the second preset usage rate.
Optionally, the first preset usage rate and the second preset usage rate may be flexibly adjusted according to specific requirement conditions of resource recovery, for example, the first preset usage rate for the log disk may be 30%, and the second preset usage rate is 70%.
It should be understood that, since the resource pressure needs to be determined in this embodiment, the resource pressure also needs to be monitored in real time in this embodiment, for example, related resources such as a memory of a service layer and a usage rate of a log disk space are added to the monitoring.
Step S14: a cache refresh rate is determined based on the resource pressure.
Optionally, in this embodiment, three cache refresh rates are respectively set for resource pressure sufficient, resource normal, and resource shortage, and the rates can be divided into slow, normal, and fast.
Specifically, when the resource pressure is small, that is, the resource is sufficient, the slow cache refresh rate may be a quantitative refresh rate for the data block in the cache data; when the resource pressure is normal, that is, the resource is normal, the normal cache back-brushing rate may be to perform cache back-brushing on the data block in the cache data based on the dirty time threshold; when the resource pressure is large, namely the resource is in shortage, the fast cache back-brushing rate can be that the dirty time is not checked, and the data blocks in the cache data are cached and back-brushed based on the sorting of the back-brushing queue.
The quantitative refresh may be to perform cache refresh on a preset number of data blocks in the cache data every time a preset interval time elapses.
Optionally, the preset interval time and the preset number in this embodiment may be flexibly set according to the specific requirement of the cache refresh.
The normal cache refresh rate may be that the dirty time is determined according to the current time and the dirty timestamp of the data block in the cache data, and the cache refresh is performed on the data block in the cache data whose dirty time exceeds the dirty time threshold.
The dirty timestamp is recorded when the specified data block associated with the date is dirty.
Optionally, since the resource pressure is large, that is, the resource is in shortage, the data blocks in the cache data are cached and flushed based on the sorting of the flushing queue, in this embodiment, the sorting of the data blocks in the flushing queue may be arranged based on the log order, that is, the log ID, and when the caching and flushing are performed, the caching and flushing are performed based on the increasing order of the log order corresponding to the data blocks.
Specifically, each log may modify the corresponding data block, that is, the designated log may modify the designated data block (object page) associated with the designated log, in this embodiment, the designated log and the designated data block are linked by a log ID, so that the designated data block corresponding to the designated log is accurately flushed when the cache is flushed.
It should be understood that, in order to enable the method to smoothly perform the above cache refresh based on the dirty time and perform sorting in the refresh queue, the method may further include adding and sorting of data blocks in the refresh queue, and specific steps thereof may refer to fig. 2, where fig. 2 is a schematic flow diagram of a refresh queue sorting step provided in an embodiment of the present application:
step S151: and when the specified data block associated with the specified log is dirty, putting the specified data block into a back-flushing queue according to the log order of the specified log.
In this embodiment, the sorting in the back-flushing queue is sorting in which the data blocks are flushed back according to the log IDs corresponding to the data blocks, and caching and back-flushing are performed with a priority of a small number of the corresponding log IDs.
Step S152: a dirty timestamp is recorded that specifies that the data block is dirty.
The dirty time stamp recorded here can be used to determine the dirty time of each data block when performing cache refresh at the normal cache refresh rate, and is compared with the dirty time threshold.
Optionally, the dirty time threshold in this embodiment may be flexibly set according to specific requirements of cache back-flushing and log recovery.
Step S16: and caching and back-brushing the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block based on the caching and back-brushing rate, wherein M and N are positive integers.
Because the data blocks associated with the logs need to be flushed before corresponding log recovery is performed, but the cache flushing sequence and the log recovery sequence are not consistent, a situation that the corresponding logs cannot be recovered for a long time after cache flushing may occur, that is, the data flushed by the cache cannot trigger corresponding log recovery. Therefore, in order to expect that each cache back-flushing can trigger effective log recovery so as to write data as much as possible on a write disk each time, in this embodiment, in addition to performing cache back-flushing on a specified data block of a specified log, data blocks before and after the corresponding specified data block are also subjected to cache back-flushing, so that the problem of log current limitation caused by the difference between the cache back-flushing sequence and the log recovery sequence is avoided.
Alternatively, M and N may be flexibly set according to the specific requirements of the cache back-brushing order and the log collection, for example, M ═ N ═ 1, M ═ N ═ 2, and the like, and further, M and N may be set to different values.
Specifically, the data block sorting in this embodiment may be identified according to a page number (pgidx), each page number corresponds to one data block, the page numbers of the data blocks may be respectively associated with corresponding log IDs, and may also be arranged in order of a small value to a large value to represent the sorting of the data blocks in the back-flushing queue.
In response, it is understood that after the cache refresh of the data block is completed, the upper layer log needs to be notified to perform resource cleaning on the designated log and the associated resource.
In order to cooperate with the cache refresh method provided in the embodiment of the present application, an embodiment of the present application further provides a cache refresh apparatus 20.
Referring to fig. 3, fig. 3 is a schematic block diagram of a cache refresh apparatus according to an embodiment of the present disclosure.
The cache refresh apparatus 20 includes:
a resource pressure determining module 21, configured to determine resource pressure, where the resource pressure includes a log disk usage rate;
a refresh rate determination module 22, configured to determine a cache refresh rate based on the resource pressure;
and the cache back-brushing module 23 is configured to perform cache back-brushing on the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block, where M and N are positive integers.
Optionally, the resource pressure determining module 21 is specifically configured to: when the utilization rate of the log disk is less than or equal to a first preset utilization rate, determining that the resource pressure is sufficient; when the utilization rate of the log disk is greater than a first preset utilization rate and less than or equal to a second preset utilization rate, determining that the resource pressure is normal; and when the utilization rate of the log disk is greater than a second preset utilization rate, determining that the resource pressure is resource shortage, wherein the first preset utilization rate is less than the second preset utilization rate.
Optionally, the back-brushing rate determining module 22 is specifically configured to: when the resources are sufficient, determining the cache back-brushing rate as the quantitative back-brushing of the data blocks in the cache data; when the resources are normal, determining the cache back-brushing rate as performing cache back-brushing on the data blocks in the cache data based on the dirty time threshold; and when the resources are in shortage, determining the cache back-brushing rate as not checking the dirty time, and performing cache back-brushing on the data blocks in the cache data based on the sorting of the back-brushing queue.
Optionally, the cache refresh module 23 is specifically configured to: and performing cache back-brushing on a preset number of data blocks in the cache data every time a preset interval time passes.
Optionally, the cache refresh module 23 is specifically configured to: determining dirty time according to the current time and dirty time stamps of data blocks in the cache data; and performing cache back-brushing on the data blocks in the cache data, wherein the dirty time of the data blocks exceeds the dirty time threshold.
Optionally, the cache brushing apparatus 20 further includes: the dirty setting module is used for putting the specified data blocks into a back-brushing queue according to the log sequence of the specified log when the specified data blocks associated with the specified log are dirty; a dirty timestamp is recorded that specifies that the data block is dirty.
Optionally, the cache brushing apparatus 20 further includes: and the notification module is used for notifying the upper-layer equipment to clean the designated log in the log disk and the associated resources of the designated log.
The embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the memory stores program instructions, and when the processor reads and runs the program instructions, the processor executes the steps in any one of the cache refresh methods provided in this embodiment.
It should be understood that the electronic device may be a Personal Computer (PC), a tablet PC, a smart phone, a Personal Digital Assistant (PDA), or other electronic device having a logical computing function.
The embodiment of the application also provides a readable storage medium, wherein the readable storage medium stores computer program instructions, and the computer program instructions are read by a processor and executed to execute the steps in the cache refresh method.
To sum up, the embodiment of the present application provides a cache refresh method, an apparatus, an electronic device, and a storage medium, where the method includes: determining resource pressure, wherein the resource pressure comprises log disk usage; determining a cache refresh rate based on the resource pressure; and caching and back-brushing the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block based on the caching and back-brushing rate, wherein M and N are positive integers.
In the implementation mode, the resource recovery is associated with the resource pressure, the rear-end I/O aggregation rate is adjusted according to the resource pressure, the throughput requirement when the resource pressure is high and the I/O aggregation efficiency when the resource pressure is low can be guaranteed, and meanwhile, the log resource recovery is effectively triggered by each back-flushing according to the sequence of the data blocks associated with the logs, so that the recovery efficiency of the log resources is stabilized.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. The apparatus embodiments described above are merely illustrative, and for example, the block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices according to various embodiments of the present application. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams, and combinations of blocks in the block diagrams, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Therefore, the present embodiment further provides a readable storage medium, in which computer program instructions are stored, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the steps of any of the block data storage methods. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a RanDom Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A cache memory back-flushing method, characterized in that the method comprises:
determining resource pressure, wherein the resource pressure comprises log disk usage;
determining a cache refresh rate based on the resource pressure;
and caching and back-brushing the specified data block associated with the specified log in the back-brushing queue, and the front M data block and the rear N data block of the specified data block based on the caching and back-brushing rate, wherein M and N are positive integers.
2. The method of claim 1, wherein the determining resource pressure comprises:
when the utilization rate of the log disk is less than or equal to a first preset utilization rate, determining that the resource pressure is sufficient;
when the utilization rate of the log disk is greater than the first preset utilization rate and less than or equal to a second preset utilization rate, determining that the resource pressure is normal;
and when the utilization rate of the log disk is greater than the second preset utilization rate, determining that the resource pressure is resource tension, wherein the first preset utilization rate is less than the second preset utilization rate.
3. The method of claim 2, wherein determining a cache back-brushing rate based on the resource pressure comprises:
when the resources are sufficient, determining the cache back-brushing rate as quantitative back-brushing of the data blocks in the cache data;
when the resources are normal, determining the cache back-brushing rate as performing cache back-brushing on the data blocks in the cache data based on a dirty time threshold;
and when resources are in shortage, determining the cache back-brushing rate as the dirty time is not checked, and performing cache back-brushing on the data blocks in the cache data based on the sorting of the back-brushing queue.
4. The method of claim 3, wherein the quantitatively scrubbing back the data blocks in the cached data comprises:
and performing cache back-flushing on a preset number of data blocks in the cache data every time a preset interval time passes.
5. The method of claim 3, wherein the flushing back the cache of the data block in the cache data based on the dirty time threshold comprises:
determining dirty time according to the current time and the dirty time stamp of the data block in the cache data;
and performing cache back-brushing on the data blocks in the cache data, wherein the dirty time of the data blocks exceeds the dirty time threshold.
6. The method of any of claims 1-5, wherein prior to the cache-back flushing the specified data block associated with the specified log and the first M and last N data blocks of the specified data block in the back-flush queue based on the cache-back flushing rate, the method further comprises:
when the designated data block associated with the designated log is dirty, putting the designated data block into the back-flushing queue according to the log order of the designated log;
recording a dirty timestamp that the specified data block is dirty.
7. The method of claim 1, wherein after the cache-back flushing of the specified data block associated with the specified log and the first M and last N data blocks of the specified data block in the back-flush queue based on the cache-back flushing rate, the method further comprises:
and informing the upper layer equipment to clean the designated log in the log disk and the associated resources of the designated log.
8. A cache memory refresh apparatus, the apparatus comprising:
the resource pressure determining module is used for determining resource pressure, and the resource pressure comprises log disk utilization rate;
the back-brushing rate determining module is used for determining the cache back-brushing rate based on the resource pressure;
and the cache back-brushing module is used for performing cache back-brushing on the specified data block associated with the specified log in the back-brushing queue, the front M data block and the rear N data block of the specified data block based on the cache back-brushing rate, wherein M and N are positive integers.
9. An electronic device comprising a memory having stored therein program instructions and a processor that, when executed, performs the steps of the method of any of claims 1-7.
10. A storage medium having stored thereon computer program instructions for executing the steps of the method according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110742870.6A CN113434296B (en) | 2021-07-01 | 2021-07-01 | Cache back-brushing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110742870.6A CN113434296B (en) | 2021-07-01 | 2021-07-01 | Cache back-brushing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113434296A true CN113434296A (en) | 2021-09-24 |
| CN113434296B CN113434296B (en) | 2023-03-21 |
Family
ID=77758549
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110742870.6A Active CN113434296B (en) | 2021-07-01 | 2021-07-01 | Cache back-brushing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113434296B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020108017A1 (en) * | 2001-02-05 | 2002-08-08 | International Business Machines Corporation | System and method for a log-based non-volatile write cache in a storage controller |
| CN102945278A (en) * | 2012-11-09 | 2013-02-27 | 华为技术有限公司 | Method and device for redoing logs of database records |
| CN108334277A (en) * | 2017-05-10 | 2018-07-27 | 中兴通讯股份有限公司 | A kind of daily record write-in and synchronous method, device, system, computer storage media |
| CN109614344A (en) * | 2018-12-12 | 2019-04-12 | 浪潮(北京)电子信息产业有限公司 | A kind of spatial cache recovery method, device, equipment and storage system |
| CN109960686A (en) * | 2019-03-26 | 2019-07-02 | 北京百度网讯科技有限公司 | The log processing method and device of database |
| CN111694703A (en) * | 2019-03-13 | 2020-09-22 | 阿里巴巴集团控股有限公司 | Cache region management method and device and computer equipment |
| CN111796996A (en) * | 2020-07-03 | 2020-10-20 | 深圳大普微电子科技有限公司 | Solid state disk log processing method, solid state disk and electronic equipment |
| CN112416261A (en) * | 2020-12-11 | 2021-02-26 | 上海中通吉网络技术有限公司 | Data storage method, device, equipment and system and storage medium |
-
2021
- 2021-07-01 CN CN202110742870.6A patent/CN113434296B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020108017A1 (en) * | 2001-02-05 | 2002-08-08 | International Business Machines Corporation | System and method for a log-based non-volatile write cache in a storage controller |
| CN102945278A (en) * | 2012-11-09 | 2013-02-27 | 华为技术有限公司 | Method and device for redoing logs of database records |
| CN108334277A (en) * | 2017-05-10 | 2018-07-27 | 中兴通讯股份有限公司 | A kind of daily record write-in and synchronous method, device, system, computer storage media |
| CN109614344A (en) * | 2018-12-12 | 2019-04-12 | 浪潮(北京)电子信息产业有限公司 | A kind of spatial cache recovery method, device, equipment and storage system |
| CN111694703A (en) * | 2019-03-13 | 2020-09-22 | 阿里巴巴集团控股有限公司 | Cache region management method and device and computer equipment |
| CN109960686A (en) * | 2019-03-26 | 2019-07-02 | 北京百度网讯科技有限公司 | The log processing method and device of database |
| CN111796996A (en) * | 2020-07-03 | 2020-10-20 | 深圳大普微电子科技有限公司 | Solid state disk log processing method, solid state disk and electronic equipment |
| CN112416261A (en) * | 2020-12-11 | 2021-02-26 | 上海中通吉网络技术有限公司 | Data storage method, device, equipment and system and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113434296B (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8548948B2 (en) | Methods and apparatus for a fine grained file data storage system | |
| Wires et al. | Characterizing storage workloads with counter stacks | |
| CN109960686B (en) | Log processing method and device for database | |
| US11947829B2 (en) | Data writing method, device, storage server, and computer readable storage medium | |
| US9177028B2 (en) | Deduplicating storage with enhanced frequent-block detection | |
| CN101419564B (en) | Method and device for recovering data by employing snapshot | |
| JP2003015912A (en) | System and method for control of fragmentation in message logging | |
| CN108959526B (en) | Log management method and log management device | |
| CN107506266B (en) | Data recovery method and system | |
| CN111857589A (en) | SSD cache down-flushing speed control method and system in distributed storage system | |
| CN113434296B (en) | Cache back-brushing method and device, electronic equipment and storage medium | |
| CN104376094A (en) | File hierarchical storage method and system taking visit randomness into consideration | |
| EP3588913A1 (en) | Data caching method and apparatus | |
| CN103236938A (en) | Method and system for user action collection based on cache memory and asynchronous processing technology | |
| CN113268437A (en) | Method and equipment for actively triggering memory sorting | |
| CN113254270B (en) | Self-recovery method, system and storage medium for storing cache hot spot data | |
| CN113806087B (en) | Method and device for adjusting service speed based on brushing speed | |
| CN112905121B (en) | Data refreshing method and system | |
| CN115658382A (en) | Incremental object backup recovery method and system for object storage and storage medium | |
| CN109240604B (en) | RAID 5-based method and system for writing user data in solid state disk | |
| CN110795034B (en) | Data migration method, device and equipment of storage system and readable storage medium | |
| CN112328539A (en) | Data migration method based on big data | |
| CN109947365B (en) | Distributed storage data verification method and device | |
| CN113076062A (en) | Method and equipment for prolonging service life of QLCSSD | |
| WO2016060700A1 (en) | File system journaling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |