CN113254270B - Self-recovery method, system and storage medium for storing cache hot spot data - Google Patents
Self-recovery method, system and storage medium for storing cache hot spot data Download PDFInfo
- Publication number
- CN113254270B CN113254270B CN202110589628.XA CN202110589628A CN113254270B CN 113254270 B CN113254270 B CN 113254270B CN 202110589628 A CN202110589628 A CN 202110589628A CN 113254270 B CN113254270 B CN 113254270B
- Authority
- CN
- China
- Prior art keywords
- storage node
- cache
- data
- storage
- hot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011084 recovery Methods 0.000 title claims abstract description 116
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 31
- 230000008569 process Effects 0.000 description 18
- 238000012544 monitoring process Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 235000019633 pungent taste Nutrition 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012896 Statistical algorithm Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1469—Backup restoration techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a self-recovery method, a self-recovery system and a storage medium for storing cache hotspot data, wherein the method comprises the following steps: the storage node carries out heat statistics on the data blocks through a heat algorithm, and the data blocks are sorted according to heat statistical parameters to form a heat queue for counting data block information; the storage node writes the content of the hot queue into a recovery database; after the storage equipment is recovered from the fault, the storage node reads the content of the hot queue of the storage node from the recovery database; and the storage node indexes the data block according to the content of the hot queue and pulls up the data block data to the storage node cache equipment. The method is realized through a self-recovery system and a storage medium for storing the cache hot spot data. According to the method and the device, the hot queue content of the storage node can be stored in the recovery database, when the hot data is lost due to cache faults, the hot queue content is recovered to the storage node through the recovery database, and the data is pulled up to the cache device according to the hot queue content, so that the hot data can be cached quickly.
Description
Technical Field
The present application relates to the field of storage system cache hotspot data recovery methods, and in particular, to a storage system cache hotspot data self-recovery method, system and storage medium.
Background
In view of cost, the storage system generally employs a low-speed storage medium, represented by HDD, which has a large single-disk capacity and is inexpensive but has relatively poor read/write performance, and a high-speed storage medium, represented by SSD and NVME, which has a small single-disk capacity and is inexpensive and has excellent read/write performance, as a cache layer. After a user access request is sent to a storage system, whether data of the user access request exists in a cache layer or not is judged firstly, if the data of the user access request exists, the data are directly fed back from the cache layer, if the data do not exist, the data need to be fed back through an HDD with low read-write performance, the cache layer adopts a high-speed storage medium with high read-write performance, and after the cache layer is arranged, the high-speed read-write performance of the cache layer can be fully utilized, so that the storage system provides quick data access experience for users. The storage system generally pulls up data with frequent user access and large access amount to the cache layer through a heat statistical algorithm, and the user can directly obtain a response from the cache layer when accessing the data.
However, if the storage medium of the storage system fails to cause data loss, the data lost in the cache layer is divided into two types, one type of data is distributed storage data (the nodes of the storage system cooperate with each other to store data), and the data can be recovered through a multi-copy or erasure code algorithm, and the other type of data is non-distributed data, and the data is related to the main storage node itself, and the data (such as the heat data and the heat information of the storage nodes stored in the cache) lacks a data recovery mechanism, and the data loss cannot be recovered through the help of other storage nodes. After the hardware of the cache layer of the fault storage system is replaced, because the cache data cannot be recovered, a user request needs a read-write response of the low-speed storage medium, the read-write performance of the low-speed storage medium is poor, and a heat counting algorithm needs a long time to carry out heat counting on the data block of the low-speed storage medium, determine the data block with high heat and then cache the synchronous heat data. For the user, the user's request to respond to the user experience through the low-speed storage medium at this stage is very poor; for the storage node of the storage system, if a large-scale user sends a request to a low-speed storage medium at the same time, the read-write performance bottleneck of the storage system is easily reached, and the storage node is crashed. Thus, after a storage system cache failover, the hot-point data needs to be quickly restored in the cache.
After the cache equipment fails, the access behavior of the user still can affect the data popularity, and the popularity queue content of the application can still be counted according to the access behavior of the user through a popularity algorithm after the cache equipment fails, and is issued to the database under the changed condition. And ensuring that the heat data restored to the cache equipment is the latest heat data.
Disclosure of Invention
To solve the above technical problem or at least partially solve the above technical problem, it is necessary to quickly restore the hot-point data in the cache after the storage system cache is repaired.
In a first aspect, the present application provides a self-recovery method for storing cached hotspot data, including:
the storage node carries out heat statistics on the data blocks through a heat algorithm, and the data blocks are sorted according to heat statistical parameters to form a heat queue for counting data block information;
the storage node writes the content of the hot queue into a recovery database;
after the storage node cache equipment is recovered from the fault, the storage node reads the content of the storage node hot queue from the recovery database;
and the storage node indexes the data block according to the content of the hot queue and pulls up the data block data to the storage node cache equipment.
Furthermore, the content of the hot queue comprises hot statistical parameters of the data blocks calculated by the hot algorithm, a globally unique number of the data blocks and an index of permanent storage positions of the data blocks in the storage system.
Still further, the storing node writing the contents of the heat queue to the recovery database includes:
the storage node monitors whether the content of the hot queue changes or not;
if the content of the hot queue is detected to be changed, writing the changed content of the hot queue into the recovery database in an incremental mode;
and the recovery database records the content writing time of each group of hot queues.
Further, the step of the storage node reading the contents of the hot queue of the storage node from the recovery database comprises:
configuring a unique cache device number for the cache device of the storage node;
the storage file of the recovery database is created according to the number of the cache device, and the storage file is used for recording the content of the corresponding cache device hot queue;
the storage node sends a hot queue content request to a recovery database, the hot queue content request comprises a cache device number corresponding to the fault cache device, and the recovery database feeds back the content in the corresponding storage file according to the cache device number.
Further, configuring a unique cache device number for the cache device of the storage node includes:
a cache device number library is constructed in a recovery database, and the cache device number library stores cache device numbers to be distributed;
the storage node sends a cache device number request to the recovery database, and the recovery database acquires the cache device number to be distributed according to the cache number request content and configures a unique cache device number for the cache device of the storage node;
and the recovery database creates a storage file according to the allocated cache equipment number.
Furthermore, the request for the number of the cache device includes information of the cache device of the storage node, and the recovery database selects a corresponding number of the cache device to be allocated according to the information of the cache device of the storage node.
Furthermore, when the storage node is removed, the recovery database recovers the cache device number occupied by the storage node to the cache device number library, and retrieves and deletes the corresponding storage file according to the cache device number occupied by the storage node.
Furthermore, the storage node acquires the index of the permanent storage position of the data block in the storage system from the hot queue, and pulls up the data of the corresponding storage position to the storage node cache device until the pulled-up data amount reaches a set threshold value.
In a second aspect, the present application provides a self-recovery system for storing cached hotspot data, including a storage unit, where the storage unit includes at least one storage node; the storage node carries out heat statistics on the data blocks through a heat algorithm, and the data blocks are sorted according to heat statistical parameters to form a heat queue for counting data block information;
the storage system unit is connected with a database node, and the database node is configured with a recovery database; the recovery database is used for storing the contents of the heat queue of the storage node and feeding the contents of the heat queue back to the storage node according to the heat queue content request of the storage node;
and the storage node pulls up the corresponding data block data to the cache equipment according to the content of the hot queue fed back by the recovery database.
In a third aspect, the present application provides a self-recovery storage medium for storing cached hot spot data, where the storage medium stores at least one instruction, and the instruction is executed to implement the self-recovery method for storing cached hot spot data.
Compared with the prior art, the technical scheme provided by the embodiment of the application has the following advantages:
the self-recovery method for storing the cache hotspot data provided by the application calculates the heat statistical parameters of the data blocks according to a heat algorithm; the hotness statistic parameter value represents the likelihood that the user will access the content of the data block for a period of time in the future. And counting data block information through the heat queue, wherein the data block information comprises heat counting parameters of the data block calculated by a heat algorithm, a global unique number of the data block and an index of a permanent storage position of the data block in the storage system. And when the content of the hot queue changes, synchronizing the content of the hot queue to the recovery database, so that the recovery database can always store the latest data block information of the storage node.
Once the cache device of the storage node is damaged due to the failure of the inequality, the hotspot data stored in the cache device and accessed by the user with high possibility in the future is lost. After the cache equipment is maintained, the storage node sends the cache equipment number corresponding to the maintained cache equipment to the recovery database, and the recovery database feeds back the content of the hot queue to the storage node according to the cache equipment number. And the storage node locally pulls up the data block data to the maintained cache equipment according to the content of the hot queue.
According to the method and the device, the data blocks with high access possibility of the user can be pulled up to the cache device according to the heat queues formed by sorting the heat statistical parameters, and the high read-write performance of the cache device provides rapid data service for the user.
After the cache equipment is damaged and repaired, the storage node can quickly pull up the heat data to the cache equipment according to the heat queue content obtained from the recovery database. Compared with a mode that the heat statistic parameters of the data blocks are recalculated through the heat algorithm and then the heat data are pulled up to the cache device according to the heat statistic parameters, the method and the device save recalculation time of the heat algorithm and enable the heat data of the cache device to be quickly recovered. For a user, when data is read, the recovered cache device can be utilized more quickly, the reading performance is better, and the user experience is better.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flowchart of a self-recovery method for storing cached hotspot data according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating a storage node sending contents of a hot queue to a recovery database according to an embodiment of the present application;
fig. 3 is a flowchart for providing a configuration of numbers of cache devices according to an embodiment of the present application;
FIG. 4 is a flowchart providing a method for a storage node to obtain contents of a hot queue from a recovery database according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a self-recovery system for storing cached hot spot data according to an embodiment of the present application.
The reference numbers and meanings in the figures are as follows:
1. storage unit, 2, database node.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Example 1
Referring to fig. 1, in the present embodiment, a self-recovery method for storing cache hot spot data is provided, where the self-recovery method for storing cache hot spot data includes:
s100, the storage node carries out heat statistics on the data blocks through a heat algorithm and sorts the data blocks according to heat statistical parameters to form a heat queue for counting data block information; the hotness algorithm in this application includes, but is not limited to, lru (least recent used) algorithm, lfu (least frequency used) algorithm. The LRU algorithm is an algorithm for predicting future frequently-accessed hot data in a cache based on access time, and the basic principle is that the more the recently frequently-accessed data is accessed in the future, a doubly-linked list is used for maintaining the recently-accessed sequence of the data, and relatively-non-accessed data is eliminated when the data needs to be eliminated. The LFU is a hot data algorithm which predicts that the data in the cache is likely to be frequently accessed in the future based on the access frequency, and the basic principle is that the data which is considered to be accessed most frequently is most likely to be accessed in the future; and maintaining a counter, counting the counter according to the accessed frequency and the access interval time of the data, sequencing the count of the counter, and eliminating the counter with relatively low count when the data needs to be eliminated. The method comprises the following steps of counting the hot degree of a data block according to the access behavior of a user by using a hot degree algorithm, wherein the access behavior comprises accessing a cache device and accessing a disk. Therefore, the hot degree algorithm continuously counts the hot degree of the data block based on the access behavior of the user to the disk after the cache equipment fails; the hot queue can still be updated.
The heat statistical parameter in the application represents the amount of representing the heat of the data block calculated by the heat algorithm. The data block in the application is a basic storage unit distributed and read by a storage system.
The content of the hot queue in the application comprises hot statistical parameters of the data blocks calculated by a hot algorithm, global unique numbers of the data blocks and indexes of permanent storage positions of the data blocks in a storage system. The distributed storage system is composed of a plurality of storage nodes, data blocks of the storage nodes are configured with global unique numbers to distinguish different data blocks, and indexes of permanent storage positions of the data blocks in the storage system are used for positioning paths of the data blocks.
S200, the storage node writes the content of the heat queue into a recovery database; in a specific implementation, referring to fig. 2, the writing of the contents of the hot queue into the recovery database by the storage node includes:
s201, a storage node monitors whether the content of a heat queue changes or not; specifically, the storage node monitors the change of the content of the hot queue caused by the elimination of the old hot data and the addition of the new hot data through a monitoring process.
S202, if the content of the hot queue is detected to be changed, writing the changed content of the hot queue into the recovery database in an incremental mode; in the specific implementation process, when the monitoring process monitors that the content of the heat queue changes, the process for uploading the content of the heat queue is awakened, and the process for uploading the content of the heat queue uploads the current heat queue to the recovery database and stores the current heat queue in an incremental manner.
S203, if the uploading is successful, the recovery database feeds back a message to the storage node, and if the uploading is failed, the recovery database feeds back a failure message to the storage node;
and S204, when the process fails, the heat queue content uploading process uploads the heat queue content until the set times or success is reached.
In a specific implementation process, the recovery database records the content writing time of each group of the hot queues.
S300, after the storage node cache equipment is recovered from the fault, the storage node reads the hot queue content of the storage node from the recovery database.
The storage system may include a plurality of storage nodes, and the storage nodes may be provided with a plurality of sets of cache devices, so that a plurality of sets of hot queue contents may be stored in the recovery database.
Specifically, a cache device number library is constructed in a recovery database, the cache device number library stores cache device numbers to be distributed, and the cache device numbers stored in the cache device number library are not repeated; in a specific implementation process, the cache numbers to be allocated in the cache device number library determine an allocation sequence according to the size of the numerical value, and one feasible way is to allocate the numerical value smaller than the cache numbers to be allocated preferentially.
Referring to fig. 3, the process of sending the number of the cache device to the storage node by the recovery database includes:
the storage system is connected with a recovery database, each storage node of the storage system sends a cache equipment number request to the recovery database, and the cache equipment number request comprises storage node cache equipment information;
after receiving a cache device number request, a recovery database selects a corresponding cache device number to be distributed according to the storage node cache device information, and establishes a mapping relation between the storage node cache device and the cache device number for storage;
and the recovery database feeds the mapping relation back to the storage node, so that the recovery database acquires the number of the cache equipment to be allocated according to the cache number request and configures the unique cache equipment number for the cache equipment of the storage node.
The recovery database creates a storage file according to the allocated cache equipment number, and the storage file is used for recording the corresponding cache equipment hot queue content; in a specific implementation process, the recovery database respectively constructs data tables by taking the numbers of the cache devices as names, and the data tables are used for storing the contents of the hot queues of the corresponding cache devices.
When the storage node is removed from the storage system, the recovery database recovers the cache device number occupied by the storage node to the cache device number library, and retrieves and deletes the corresponding storage file according to the cache device number occupied by the storage node. Specifically, the removed storage node sends a unbinding request to the recovery database, where the unbinding request includes a number of the cache device occupied by the storage node. On one hand, the recovery database recovers the cache equipment number occupied by the storage node into a cache equipment number library; and on the other hand, the recovery database retrieves and deletes the corresponding storage file according to the number of the cache device occupied by the storage node.
On this basis, referring to fig. 4, the process of the storage node reading the contents of the hot queue of the storage node from the recovery database is as follows:
s301, a storage node sends a hot queue content request to a recovery database, wherein the hot queue content request comprises a cache device number corresponding to a fault cache device;
s302, the recovery database inquires corresponding storage files according to the cache equipment numbers in the hot queue content request and obtains the contents in the storage files;
s303, the recovery database feeds back the content in the corresponding storage file to the storage node.
S400, the storage node indexes the data blocks according to the content of the hot queue and pulls up the data blocks to the storage node cache device. And the storage node acquires the index of the permanent storage position of the data block in the storage system from the hot queue, and pulls up the data of the corresponding storage position to the storage node cache equipment until the pulled-up data amount reaches a set threshold value. When a user makes a data request, the storage system firstly checks whether the request is that the data is in the cache device, and if the request data is directly fed back to the user through the cache device, the cache device provides high-performance reading and writing experience for the user.
Example 2
The application provides a self-recovery system for storing cache hot spot data.
Referring to fig. 5, the self-recovery system for storing cached hot spot data includes a storage unit, where the storage unit includes at least one storage node; the storage nodes carry out heat statistics on the data blocks through a heat algorithm, and a heat queue for counting data block information is formed according to the heat statistical parameter sequence obtained by the heat algorithm; the storage node is provided with a monitoring process, and the monitoring process monitors the change of the content of the heat queue caused by the elimination of the old heat data and the addition of the new heat data; the storage node is also provided with a hot queue content uploading process, and when the monitoring process monitors that the hot queue content changes, the hot queue content uploading process is awakened; the storage node is configured with a cache device information acquisition process, and the cache device information acquisition process is used for acquiring cache device information of the storage node.
The storage system unit is connected with a database node, and the database node is configured with a recovery database; on one hand, the recovery database is used for storing the contents of the hot queue uploaded by the storage node through a hot queue content uploading process, and on the other hand, the recovery database feeds the contents of the hot queue back to the storage node according to the hot queue content request of the storage node;
and the storage node pulls up the corresponding data block data to the cache equipment according to the content of the hot queue fed back by the recovery database.
Example 3
The application provides a self-recovery storage medium for storing cache hot spot data. The self-recovery storage medium for storing the cache hotspot data stores at least one instruction, and the instruction is executed to realize the self-recovery method for the cache hotspot data.
The principle advantages of the technical scheme provided by the application are as follows:
the self-recovery method for storing the cache hotspot data provided by the application calculates the heat statistical parameters of the data blocks according to a heat algorithm; the hotness statistic parameter value represents the likelihood that the user will access the content of the data block for a period of time in the future. And counting data block information through the heat queue, wherein the data block information comprises heat counting parameters of the data block calculated by a heat algorithm, a global unique number of the data block and an index of a permanent storage position of the data block in the storage system. And when the content of the heat queue changes, synchronizing the content of the heat queue to the recovery database, so that the recovery database can always store the latest data block information of the storage node.
Once the cache device of the storage node is damaged due to the failure of the inequality, the hotspot data stored in the cache device and accessed by the user with high possibility in the future is lost. After the cache equipment is maintained, the storage node sends the cache equipment number corresponding to the maintained cache equipment to the recovery database, and the recovery database feeds back the content of the hot queue to the storage node according to the cache equipment number. And the storage node locally pulls up the data block data to the maintained cache equipment according to the content of the hot queue.
According to the method and the device, the data blocks with high access possibility of the user can be pulled up to the cache device according to the heat queues formed by sorting the heat statistical parameters, and the high read-write performance of the cache device provides rapid data service for the user.
After the cache equipment is damaged and repaired, the storage node can quickly pull up the heat data to the cache equipment according to the heat queue content obtained from the recovery database. Compared with a mode that the heat degree statistical parameters of the data blocks are recalculated through the heat degree algorithm and then the heat degree data are pulled up to the cache device according to the heat degree statistical parameters, the method and the device save recalculation time of the heat degree algorithm and enable the heat degree data of the cache device to be quickly recovered. For a user, when data is read, the recovered cache device can be utilized more quickly, the reading performance is better, and the user experience is better.
In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A self-recovery method for storing cache hotspot data is applied to a storage node configured with a cache layer, the cache layer is composed of at least one cache device, and the cache layer stores data block data for a user to access quickly, and is characterized by comprising the following steps:
the storage node carries out heat statistics on the data blocks through a heat algorithm, and the data blocks are sorted according to heat statistical parameters to form a heat queue for counting data block information;
the storage node writes the content of the hot queue into a recovery database;
after the storage node cache equipment is recovered from the fault, the storage node reads the content of the storage node hot queue from the recovery database;
and the storage node indexes the data block according to the content of the hot queue and pulls up the data block data to the storage node cache equipment.
2. The self-recovery method for data in a storage cache hotspot according to claim 1, wherein the content of the hot queue comprises a hot statistical parameter of the data block calculated by a hot algorithm, a globally unique number of the data block, and an index of a permanent storage position of the data block in the storage system.
3. The self-recovery method for storing cached hot spot data according to claim 1, wherein the step of writing the contents of the hot queue into the recovery database by the storage node comprises:
the storage node monitors whether the content of the hot queue changes or not;
if the content of the hot queue is detected to be changed, writing the changed content of the hot queue into the recovery database in an incremental mode;
and the recovery database records the content writing time of each group of hot queues.
4. The storage cache hot spot data self-recovery method according to claim 1, wherein the step of reading the contents of the hot queue of the storage node from the recovery database by the storage node comprises:
configuring a unique cache device number for the cache device of the storage node;
the storage file of the recovery database is created according to the number of the cache device, and the storage file is used for recording the content of the corresponding cache device hot queue;
the storage node sends a hot queue content request to a recovery database, the hot queue content request comprises a cache device number corresponding to the fault cache device, and the recovery database feeds back the content in the corresponding storage file according to the cache device number.
5. The self-recovery method for storing cached hot spot data according to claim 4, wherein configuring a unique caching device number for the caching device of the storage node comprises:
a cache device number library is constructed in a recovery database, and the cache device number library stores cache device numbers to be distributed;
the storage node sends a cache device number request to the recovery database, and the recovery database acquires the cache device number to be distributed according to the cache number request content and configures a unique cache device number for the cache device of the storage node;
and the recovery database creates a storage file according to the allocated cache equipment number.
6. The self-recovery method for storing cached hot spot data according to claim 5, wherein the request for the number of the caching device comprises information of the caching device of the storage node, and the recovery database selects a corresponding number of the caching device to be allocated according to the information of the caching device of the storage node.
7. The self-recovery method for storing cached hot spot data according to claim 5, wherein when a storage node is removed, the recovery database recovers the number of the caching device occupied by the storage node to the caching device number library, and retrieves and deletes the corresponding storage file according to the number of the caching device occupied by the storage node.
8. The self-recovery method for hot spot data in storage cache according to claim 2, wherein the storage node obtains the index of the permanent storage location of the data block in the storage system from the hot queue, and pulls up the data in the corresponding storage location to the storage node cache device until the amount of the pulled-up data reaches a set threshold.
9. The self-recovery system for the storage cache hot spot data is characterized by comprising a storage unit, wherein the storage unit comprises at least one storage node; the storage node carries out heat statistics on the data blocks through a heat algorithm, and the data blocks are sorted according to heat statistical parameters to form a heat queue for counting data block information;
the storage system unit is connected with a database node, and the database node is configured with a recovery database; the recovery database is used for storing the content of the hot queue of the storage node and feeding back the content of the hot queue to the storage node according to the hot queue content request of the storage node;
and the storage node pulls up the corresponding data block data to the cache equipment according to the content of the hot queue fed back by the recovery database.
10. A self-recovery storage medium for storing cached hot spot data, wherein at least one instruction is stored, and the instruction is executed to implement the self-recovery method for storing cached hot spot data according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110589628.XA CN113254270B (en) | 2021-05-28 | 2021-05-28 | Self-recovery method, system and storage medium for storing cache hot spot data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110589628.XA CN113254270B (en) | 2021-05-28 | 2021-05-28 | Self-recovery method, system and storage medium for storing cache hot spot data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113254270A CN113254270A (en) | 2021-08-13 |
| CN113254270B true CN113254270B (en) | 2022-06-14 |
Family
ID=77185202
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110589628.XA Active CN113254270B (en) | 2021-05-28 | 2021-05-28 | Self-recovery method, system and storage medium for storing cache hot spot data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113254270B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113821176B (en) * | 2021-09-29 | 2023-07-21 | 重庆紫光华山智安科技有限公司 | Data migration processing method, device and storage medium |
| CN117992461A (en) * | 2024-04-03 | 2024-05-07 | 深圳九有数据库有限公司 | Database data storage method based on column-type memory storage mode |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7146521B1 (en) * | 2002-08-21 | 2006-12-05 | 3Pardata, Inc. | Preventing damage of storage devices and data loss in a data storage system |
| CN105045685A (en) * | 2015-08-04 | 2015-11-11 | 浪潮电子信息产业股份有限公司 | Backup method and apparatus for cached data and recovery method and apparatus for cached data |
| CN105740172A (en) * | 2016-01-26 | 2016-07-06 | 浪潮(北京)电子信息产业有限公司 | Block device data cache power-down protection method and system |
| CN106527988A (en) * | 2016-11-04 | 2017-03-22 | 郑州云海信息技术有限公司 | SSD (Solid State Drive) data migration method and device |
| CN109947787A (en) * | 2017-10-30 | 2019-06-28 | 阿里巴巴集团控股有限公司 | A kind of storage of data hierarchy, hierarchical query method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109800260A (en) * | 2018-12-14 | 2019-05-24 | 深圳壹账通智能科技有限公司 | High concurrent date storage method, device, computer equipment and storage medium |
| CN112764664A (en) * | 2019-10-21 | 2021-05-07 | 深圳市茁壮网络股份有限公司 | Disk caching method and device |
-
2021
- 2021-05-28 CN CN202110589628.XA patent/CN113254270B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7146521B1 (en) * | 2002-08-21 | 2006-12-05 | 3Pardata, Inc. | Preventing damage of storage devices and data loss in a data storage system |
| CN105045685A (en) * | 2015-08-04 | 2015-11-11 | 浪潮电子信息产业股份有限公司 | Backup method and apparatus for cached data and recovery method and apparatus for cached data |
| CN105740172A (en) * | 2016-01-26 | 2016-07-06 | 浪潮(北京)电子信息产业有限公司 | Block device data cache power-down protection method and system |
| CN106527988A (en) * | 2016-11-04 | 2017-03-22 | 郑州云海信息技术有限公司 | SSD (Solid State Drive) data migration method and device |
| CN109947787A (en) * | 2017-10-30 | 2019-06-28 | 阿里巴巴集团控股有限公司 | A kind of storage of data hierarchy, hierarchical query method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113254270A (en) | 2021-08-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9355112B1 (en) | Optimizing compression based on data activity | |
| CN107943867B (en) | High-performance hierarchical storage system supporting heterogeneous storage | |
| US6675176B1 (en) | File management system | |
| CN113254270B (en) | Self-recovery method, system and storage medium for storing cache hot spot data | |
| CN109947363B (en) | Data caching method of distributed storage system | |
| CN105549905A (en) | Method for multiple virtual machines to access distributed object storage system | |
| KR20120090965A (en) | Apparatus, system, and method for caching data on a solid-state strorage device | |
| AU2013409624A1 (en) | Data processing method and device in distributed file storage system | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| US6961727B2 (en) | Method of automatically generating and disbanding data mirrors according to workload conditions | |
| CN113568582B (en) | Data management method, device and storage equipment | |
| CN113626431A (en) | LSM tree-based key value separation storage method and system for delaying garbage recovery | |
| KR20090063733A (en) | Method recovering data server at the applying multiple reproduce dispersion file system and metadata storage and save method thereof | |
| CN110908589A (en) | Data file processing method, device and system and storage medium | |
| CN111708894B (en) | Knowledge graph creation method | |
| US8239427B2 (en) | Disk layout method for object-based storage devices | |
| CN104917788A (en) | Data storage method and apparatus | |
| CN113704217A (en) | Metadata and data organization architecture method in distributed persistent memory file system | |
| CN111831691B (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN109002400B (en) | Content-aware computer cache management system and method | |
| CN102609508B (en) | High-speed access method of files in network storage | |
| CN107133334B (en) | Data synchronization method based on high-bandwidth storage system | |
| CN112463795A (en) | Dynamic hash method, device, equipment and storage medium | |
| CN116643704A (en) | Storage management method, storage management device, electronic equipment and storage medium | |
| CN116541553A (en) | Video scheduling method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |