CN112286950A - Method for keeping consistency of valid data in distributed cache - Google Patents
Method for keeping consistency of valid data in distributed cache Download PDFInfo
- Publication number
- CN112286950A CN112286950A CN202011309346.1A CN202011309346A CN112286950A CN 112286950 A CN112286950 A CN 112286950A CN 202011309346 A CN202011309346 A CN 202011309346A CN 112286950 A CN112286950 A CN 112286950A
- Authority
- CN
- China
- Prior art keywords
- cache
- task
- node
- updating
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for keeping the consistency of valid data in a distributed cache, which updates the cache data of each node of the distributed cache by arranging a cache updating message manager and a cache node updating monitoring device, monitors the updating state and removes the abnormally updated node, improves the consistency of the cache data and effectively improves the correct utilization rate of the cache data.
Description
Technical Field
The invention relates to the technical field of data caching, in particular to a method for keeping effective data consistency in distributed cache.
Background
Distributed caching is widely applied to various large systems at present. Especially, when the traffic is huge, the application cache can well support high concurrent access. However, there is a problem with either type of cache, requiring data coherency to be maintained. When database data changes, cache data needs to be updated in an effective time, due to the distributed system, a certain failure rate exists in updating each node cache, when a certain cache node is abnormally updated or fails, the data of the node becomes dirty data, and if the data is not processed at this time, service reference on the dirty data causes losses in different degrees.
Disclosure of Invention
The technical problems to be solved by the invention are how to update the cache information of each node to keep data consistency, and how to properly process some cache nodes under the condition that the data update of the cache nodes fails, so that the service request can improve the correct cache hit rate. In order to solve the technical problem, the invention provides a method for maintaining the consistency of valid data in a distributed cache, which updates the cache data of each node of the distributed cache by arranging a cache update message manager and a cache node update monitoring device, monitors the update state and removes the abnormally updated node, improves the consistency of the cache data and effectively improves the correct utilization rate of the cache data.
In order to achieve the technical effects, the invention provides the following technical scheme:
a method for maintaining the consistency of valid data in a distributed cache comprises the following steps:
(1) a setup message manager for issuing an update message notification;
(2) each cache node is provided with a cache data updating task;
(3) setting a cache node updating monitoring device;
(4) and setting a load overload configuration task, and smoothly starting the system without interrupting service.
The further technical scheme is that in the step (1), when the database data is updated, a message notice for updating the cache is issued, wherein the message notice contains a necessary updating unique identification code.
And (3) further adopting the technical scheme that each cache node in the step (2) subscribes a theme for updating the cache, triggers a local cache updating task after receiving a notification for updating the cache, and triggers a callback task after the updating is successful.
The further technical scheme is that the step (3) comprises the following steps:
a) setting a task for receiving a data updating unique identification code, subscribing a theme for updating the cache and storing data updating unique identification code information;
b) setting a cache node monitoring association table for storing the state information of the update task of each cache node;
c) setting and receiving a callback task after the updating is successful, and calling back the unique identification code information to the task and triggering the updating of the corresponding node information in the association table after the cache of a certain node is successfully updated;
d) setting a task of scanning and monitoring the association table at regular time, and checking whether the cache data of each node is updated;
e) and setting a task of removing the failed node in the load, after the task of scanning and monitoring the association table is executed, checking the task with the unique identification code stored in a subscription mode, finding the node with abnormal update, and triggering the task.
The updating process of the cache data specifically comprises the following steps: when the real data of the database is changed, the message manager is triggered to issue a message with the unique identification code of the updating task, and before this, each cache node and the cache node updating monitoring device in the system subscribe the theme message of the updating data. After the update data topic message is published, each subscribing device triggers the post task of the subscribing device, the cache node updates the cache, and the monitoring device keeps the unique identification code of the update task. And after the cache node finishes updating self cache data, the unique identification code with the task update is sent to a callback task of the monitoring device, and the monitoring device recalls the task update monitoring association table. In addition, a scanning monitoring association table is arranged at a certain time frequency, whether the updating state of each node is consistent with the unique identification code updated by the task or not is scanned, inconsistent node information is sent to the task of removing the failed node for processing, and the failed node information is removed from the system, so that loss caused by service quoting to the data which are not updated on the nodes is avoided. The configuration is then reloaded, and the system is started smoothly.
Compared with the prior art, the invention has the following beneficial effects: according to the method and the device, the cache is updated through the message publishing subscription mode, the distributed cache nodes are monitored, the cache nodes which are abnormally updated are removed, and the correct hit rate of the cache data is effectively improved. The method can effectively process the nodes with abnormal updating after the updating of the distributed cache fails, thereby improving the usability of the whole distributed cache application.
Drawings
FIG. 1 is a flow diagram of maintaining valid data coherency in a distributed cache.
Detailed Description
Example 1
To update the commercial example description.
A method for maintaining the consistency of valid data in a distributed cache comprises the following steps:
(1) the message manager MQS is set.
When some commodity information in the database is changed, an instruction is issued to the MQS, and a message is issued. The message is subject to: SUB _ UPDATE _ GOODS, the message content is: the unique identification code CodeXX + commodity updating content.
(2) Each cache node sets a cache data updating task updateGoodsCache.
Each cache node subscribes to the message with the topic of SUB _ UPDATE _ GOODS, and when the message is received, the cache node executes the task of updating cache data, namely updateGoodsCache, and UPDATEs the cached commodity data. And sending the current node information and the unique identification code to a callback task callback NodeStatus.
(3) And setting a cache node to update the monitoring device MonitorCachenodenode.
(a) And setting a task receiveCode for updating the unique identification code by the received data. Subscribing the subject SUB _ UPDATE _ GOODS message, and saving the received task unique identification code CodeXX.
(b) And setting a cache node to monitor an association table tablenodenandstatus. And the cache node is used for storing the state information of the update tasks of the cache nodes. Presetting fields: id (primary key), node (node), code (unique identification code). The node value will store the information of each node, node1, node2. The code value will store the corresponding task unique identification code.
(c) And setting a callback task callback NodeStatus after the updating is successful. For example, after the node1 node successfully updates the cache, the current unique identification Code _89SHD and the node information node1 are sent to the callback task callback node status. After the task is executed, the Code in the record with the node1 in the updated monitoring association table tableeNodeAndStatus is Code _89 SHD. By analogy, the Code corresponding to the nodeN should be Code _89SHD normally. And if the callback task does not receive the feedback state information of a certain node, recording the code as other nodes by the node.
(d) And setting a timed scanning monitoring association table task crontNodeMonitor. Once a minute (the specific time length is adjusted according to the service requirement). The task scans the code information of each node in the tableeNodeAndStatus, collates the code information with the CodeXX stored in 5.3.1, and if the code information is inconsistent with the CodeXX, the task can judge that the data of the cache node is not updated in the valid period and regards the cache node as a removable node. Such as node3 being Code _ other.
(e) And setting a removal failure node task removeNode in the load. This task will remove the received failed node3 on the associated load configuration. For example, node3 is referenced by load 1, then node3 in the node configuration of load 1 is removed.
(4) And setting a load overload configuration task reloadService. The removeNode triggers the task after the execution is completed, and reloads the load 1 configuration and then smoothly restarts the load 1 service.
Although the present invention has been described herein with reference to the illustrated embodiments thereof, which are intended to be preferred embodiments of the present invention, it is to be understood that the invention is not limited thereto, and that numerous other modifications and embodiments can be devised by those skilled in the art that will fall within the spirit and scope of the principles of this disclosure.
Claims (4)
1. A method for maintaining valid data consistency in a distributed cache, comprising the steps of:
(1) a setup message manager for issuing an update message notification;
(2) each cache node is provided with a cache data updating task;
(3) setting a cache node updating monitoring device;
(4) and setting a load overload configuration task, and smoothly starting the system without interrupting service.
2. The method for maintaining consistency of valid data in distributed caches according to claim 1, wherein in step (1), when the database data is updated, a message notification for updating the cache is issued, which contains the necessary unique identification code for updating.
3. The method for maintaining the consistency of the valid data in the distributed caches according to claim 1, wherein in the step (2), each cache node subscribes to a subject of the update cache, and after receiving a notification of the update cache, each cache node triggers a local cache update task, and after the update is successful, triggers a callback task.
4. The method for maintaining consistency of valid data in distributed caches according to claim 1, wherein said step (3) comprises the steps of:
a) setting a task for receiving a data updating unique identification code, subscribing a theme for updating the cache and storing data updating unique identification code information;
b) setting a cache node monitoring association table for storing the state information of the update task of each cache node;
c) setting and receiving a callback task after the updating is successful, and calling back the unique identification code information to the task and triggering the updating of the corresponding node information in the association table after the cache of a certain node is successfully updated;
d) setting a task of scanning and monitoring the association table at regular time, and checking whether the cache data of each node is updated;
e) and setting a task of removing the failed node in the load, after the task of scanning and monitoring the association table is executed, checking the task with the unique identification code stored in a subscription mode, finding the node with abnormal update, and triggering the task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011309346.1A CN112286950A (en) | 2020-11-20 | 2020-11-20 | Method for keeping consistency of valid data in distributed cache |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011309346.1A CN112286950A (en) | 2020-11-20 | 2020-11-20 | Method for keeping consistency of valid data in distributed cache |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112286950A true CN112286950A (en) | 2021-01-29 |
Family
ID=74398318
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011309346.1A Pending CN112286950A (en) | 2020-11-20 | 2020-11-20 | Method for keeping consistency of valid data in distributed cache |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112286950A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724314A (en) * | 2012-06-20 | 2012-10-10 | 深圳市远行科技有限公司 | Memcached client side based on metadata management |
| CN105630823A (en) * | 2014-11-04 | 2016-06-01 | 阿里巴巴集团控股有限公司 | Method, device and system for monitoring cache data based on distributed system |
| US20170344481A1 (en) * | 2016-05-31 | 2017-11-30 | Salesforce.Com, Inc. | Invalidation and refresh of multi-tier distributed caches |
| CN108540367A (en) * | 2017-03-06 | 2018-09-14 | 中国移动通信有限公司研究院 | A kind of message treatment method and system |
| CN109800239A (en) * | 2019-01-15 | 2019-05-24 | 科大国创软件股份有限公司 | Distributed structure/architecture data sharing method based on Redis |
| CN111782666A (en) * | 2020-06-19 | 2020-10-16 | 京东数字科技控股有限公司 | Cache service system |
-
2020
- 2020-11-20 CN CN202011309346.1A patent/CN112286950A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102724314A (en) * | 2012-06-20 | 2012-10-10 | 深圳市远行科技有限公司 | Memcached client side based on metadata management |
| CN105630823A (en) * | 2014-11-04 | 2016-06-01 | 阿里巴巴集团控股有限公司 | Method, device and system for monitoring cache data based on distributed system |
| US20170344481A1 (en) * | 2016-05-31 | 2017-11-30 | Salesforce.Com, Inc. | Invalidation and refresh of multi-tier distributed caches |
| CN108540367A (en) * | 2017-03-06 | 2018-09-14 | 中国移动通信有限公司研究院 | A kind of message treatment method and system |
| CN109800239A (en) * | 2019-01-15 | 2019-05-24 | 科大国创软件股份有限公司 | Distributed structure/architecture data sharing method based on Redis |
| CN111782666A (en) * | 2020-06-19 | 2020-10-16 | 京东数字科技控股有限公司 | Cache service system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6330600B1 (en) | System for synchronizing configuration information of a network element if received trap sequence number is out-of-sequence | |
| US7043727B2 (en) | Method and system for efficient distribution of network event data | |
| CN108566291B (en) | Event processing method, server and system | |
| US7912858B2 (en) | Data synchronization method | |
| US5974429A (en) | Method and apparatus for updating distributed databases in a telecommunications network | |
| CN112260876A (en) | Dynamic gateway route configuration method, platform, computer equipment and storage medium | |
| CN106909411A (en) | A kind of file updating method and device | |
| CN107623703B (en) | Synchronization method, device and system for Global Transaction Identifier (GTID) | |
| CN114764380A (en) | Distributed cluster control method and device based on ETCD | |
| CN110611603A (en) | Cluster network card monitoring method and device | |
| CN112100004A (en) | Management method and storage medium of Redis cluster node | |
| CN114629883B (en) | Service request processing method and device, electronic equipment and storage medium | |
| EP1894096A1 (en) | Apparatus, system, and method for facilitating communication between an enterprise information system and a client | |
| EP2416526B1 (en) | Task switching method, server node and cluster system | |
| US20020152052A1 (en) | Electronic apparatus for detecting an abnormal state, a method for detecting the abnormal state, a system of the electronic apparatus for detecting the abnormal state, a computer program for detecting the abnormal state and a computer-readable storage medium for recording the computer program for detecting the abnormal state | |
| CN112286950A (en) | Method for keeping consistency of valid data in distributed cache | |
| CN112214377B (en) | Equipment management method and system | |
| JPH11259326A (en) | Hot standby system, automatic re-execution method for the same and storage medium therefor | |
| CN113590379A (en) | Redis data persistence method and system | |
| WO2002001347A2 (en) | Method and system for automatic re-assignment of software components of a failed host | |
| CN111158930A (en) | Redis-based high-concurrency time-delay task system and processing method | |
| US20040236800A1 (en) | Network management controlled network backup server | |
| JP2011034175A (en) | Transaction control device, transaction processing method, and program | |
| KR100832890B1 (en) | Process obstacle lookout method and recovery method for information communication | |
| CN114629783B (en) | State monitoring method, system, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210129 |
|
| RJ01 | Rejection of invention patent application after publication |