CN115269623A - Data double-writing monitoring method, device, equipment and storage medium - Google Patents

Data double-writing monitoring method, device, equipment and storage medium Download PDF

Info

Publication number
CN115269623A
CN115269623A CN202210936957.1A CN202210936957A CN115269623A CN 115269623 A CN115269623 A CN 115269623A CN 202210936957 A CN202210936957 A CN 202210936957A CN 115269623 A CN115269623 A CN 115269623A
Authority
CN
China
Prior art keywords
data
redis cluster
redis
keyword
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210936957.1A
Other languages
Chinese (zh)
Inventor
丁锡裕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Property and Casualty Insurance Company of China Ltd
Original Assignee
Ping An Property and Casualty Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Property and Casualty Insurance Company of China Ltd filed Critical Ping An Property and Casualty Insurance Company of China Ltd
Priority to CN202210936957.1A priority Critical patent/CN115269623A/en
Publication of CN115269623A publication Critical patent/CN115269623A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1464—Management of the backup or restore process for networked environments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3096—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents wherein the means or processing minimize the use of computing system or of computing system component resources, e.g. non-intrusive monitoring which minimizes the probe effect: sniffing, intercepting, indirectly deriving the monitored data from other directly available data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
    • G06F11/324—Display of status information
    • G06F11/327—Alarm or error message display
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80—Database-specific techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/82—Solving problems relating to consistency

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Security & Cryptography (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data double-write monitoring method, a device, equipment and a storage medium, wherein the method comprises the following steps: when a data storage instruction is received, executing a writing task of respectively writing data to be stored into a first redis cluster and a second redis cluster; judging whether the first redis cluster and the second redis cluster are in a service peak period or not; when the first and second redis clusters are in a business low peak period, deriving first cache data from the first redis cluster and deriving second cache data from the second redis cluster; and when the second cache data contains data inconsistent with the first cache data, sending out alarm information. The data respectively written into different redis clusters are exported and compared at the service low peak period so as to monitor whether the data are successfully written, and the consistency of the data is ensured.

Description

Data double-writing monitoring method, device, equipment and storage medium
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a method, an apparatus, a device, and a storage medium for monitoring data double writing.
Background
With the development of internet technology, especially the deep application of financial technology, more and more financial technology companies, especially internet financial technology companies, have made more stringent requirements on data security and service stability. In order to improve the application experience of enterprises, the caching technology represented by redis is applied and practiced in large scale in various large software architectures, and the double-active high-availability technology of redis is also valued by more and more enterprises. In order to meet the requirement of dual active high availability of the redis, two sets of redis clusters with the same function need to be deployed, so that the same data writing operation needs to be performed on the two sets of redis clusters when the data writing operation is performed, so as to ensure the consistency of the data. However, over time, when an abnormal condition occurs, inconsistency may occur in data of the two sets of redis clusters, for example, data write failure due to full thread pools, data write failure due to access by an application during maintenance of a redis cluster database node, data write failure due to network abnormality, and the like, and finally, the use of a user may be affected due to the inconsistency of data in the two sets of redis clusters, which may cause loss.
Disclosure of Invention
The application provides a data double-write monitoring method, device, equipment and storage medium, which are used for solving the problem that data inconsistency between a master and a slave redis clusters in the prior art cannot be effectively monitored.
In order to solve the technical problem, the application adopts a technical scheme that: a data double-write monitoring method is provided, which comprises the following steps: when a data storage instruction is received, executing a writing task of respectively writing data to be stored into a first redis cluster and a second redis cluster; judging whether the first redis cluster and the second redis cluster are in a business peak period or not; when the first redis cluster and the second redis cluster are in a service low peak period, deriving first cache data from the first redis cluster and deriving second cache data from the second redis cluster; and when the second cache data has data inconsistent with the first cache data, sending alarm information.
As a further improvement of the present application, determining whether the first redis cluster and the second redis cluster are in a service low peak period includes: acquiring access logs of a first redis cluster and a second redis cluster, and inquiring the access amount of the first redis cluster and the second redis cluster based on the access logs; judging whether the access amount is lower than a preset access amount threshold value or not; and if so, the first redis cluster and the second redis cluster are in a low traffic peak period.
As a further improvement of the present application, after sending the alarm information, the method further includes: querying the data type and the key word of inconsistent data in the first cache data and the second cache data; and synchronizing data increment corresponding to the data type and the key word in the first redis cluster to the second redis cluster.
As a further improvement of the present application, deriving the first cache data from the first redis cluster and deriving the second cache data from the second redis cluster includes: judging whether the first redis cluster and the second redis cluster adopt an active mode or a cluster mode; if the master-slave mode is adopted, first cache data is derived from any node in the first redis cluster, and second cache data is derived from any node in the second redis cluster; and if the cluster module is the cluster module, exporting the first data of all nodes in the first redis cluster, classifying the first data according to the data type to obtain first cache data, exporting the second data of all nodes in the second redis cluster, and classifying the second data according to the data type to obtain second cache data.
As a further improvement of the present application, determining whether the first redis cluster and the second redis cluster are after the service low peak period further includes: when the first redis cluster and the second redis cluster are not in the service peak period, inquiring an operation log record of the first redis cluster, and analyzing the operation log record to obtain a keyword corresponding to the operated data; respectively acquiring first data corresponding to keywords in a first redis cluster and second data corresponding to keywords in a second redis cluster; and when the first data is inconsistent with the second data, sending out alarm information of inconsistent data corresponding to the keyword.
As a further improvement of the application, before sending out the alarm information with inconsistent data corresponding to the keyword, the method further comprises the following steps: when the first data is inconsistent with the second data, recording write-in time for writing the first data corresponding to the query keyword into a first redis cluster based on the operation log; judging whether the time interval between the writing time and the current time reaches a preset time interval or not; if yes, executing to send out alarm information of inconsistent data corresponding to the keyword; and if not, inquiring whether a target writing task corresponding to the keyword exists in the data writing tasks of the second redis cluster, and if the target writing task corresponding to the keyword does not exist, executing alarm information for sending inconsistent data corresponding to the keyword.
As a further improvement of the present application, determining whether the first redis cluster and the second redis cluster are after the service low peak period further includes: when the first redis cluster and the second redis cluster are not in a service low peak period, acquiring a first operation log record of the first redis cluster at intervals of a first preset time period, and analyzing to obtain a keyword list of data successfully written into the first redis cluster in the first preset time period; querying a second operation log record of a second redis cluster based on the keyword list, and determining whether data corresponding to each keyword in the keyword list is successfully written into the second redis cluster within a second preset time period; if the data is successfully written in, removing the keywords corresponding to the data successfully written in the second redis cluster from the keyword list; and if the data is not successfully written in, marking the keyword corresponding to the data which is not successfully written in the second redis cluster in the keyword list, and sending out alarm information.
In order to solve the above technical problem, another technical solution adopted by the present application is: provided is a data double-write monitoring device, including: the storage module is used for executing write-in tasks of respectively writing data to be stored into the first redis cluster and the second redis cluster when receiving a data storage instruction; the judging module is used for judging whether the first redis cluster and the second redis cluster are in a business peak period or not; the export module is used for exporting the first cache data from the first redis cluster and exporting the second cache data from the second redis cluster when the first redis cluster and the second redis cluster are in a service low peak period; and the first alarm module is used for sending alarm information when the second cache data contains data inconsistent with the first cache data.
In order to solve the above technical problem, the present application adopts another technical solution that: there is provided a computer device comprising a processor, a memory coupled to the processor, the memory having stored therein program instructions which, when executed by the processor, cause the processor to perform the steps of the data double write monitoring method as described in the preceding.
In order to solve the above technical problem, the present application adopts another technical solution that: a storage medium is provided that stores program instructions that enable the above-described data double-write monitoring method.
The beneficial effect of this application is: according to the data double-write monitoring method, after the write-in tasks of writing data to be stored into a first redis cluster and a second redis cluster are respectively executed, when a system is in a service low peak period, first cache data and second cache data are respectively derived from the first redis cluster and the second redis cluster, then data consistency comparison is carried out on the first cache data and the second cache data, an alarm is sent out when the data are inconsistent, and a worker is reminded of maintaining the data in time.
Drawings
FIG. 1 is a flow chart of a data double-write monitoring method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data double-write monitoring method according to a second embodiment of the present invention;
FIG. 3 is a flow chart of a data double-write monitoring method according to a third embodiment of the present invention;
FIG. 4 is a functional block diagram of a data double-write monitoring apparatus according to a first embodiment of the present invention;
FIG. 5 is a functional block diagram of a data double-write monitoring device according to a second embodiment of the present invention;
FIG. 6 is a functional block diagram of a data double-write monitoring apparatus according to a third embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a computer apparatus according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a storage medium according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
The terms "first", "second" and "third" in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any indication of the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of the feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. In the embodiment of the present application, all directional indicators (such as up, down, left, right, front, rear \8230;) are used only to explain the relative positional relationship between the components, the motion situation, etc. at a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indicator is changed accordingly. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Fig. 1 is a schematic flow chart of a data double-write monitoring method according to a first embodiment of the present invention. It should be noted that the method of the present invention is not limited to the flow sequence shown in fig. 1 if the results are substantially the same. As shown in fig. 1, the method comprises the steps of:
step S101: and when a data storage instruction is received, executing a writing task of respectively writing the data to be stored into the first redis cluster and the second redis cluster.
It should be understood that the redis cluster is a distributed storage system, which fragments data and stores different data on different nodes, thereby solving the storage problem of mass data. In this embodiment, in order to meet the requirement of dual-active high availability of the system, two sets of first and second redis clusters having the same function are set, each of the first and second redis clusters adopts a master-slave architecture, and when data write operation is performed, the same data write operation needs to be performed on the first and second redis clusters to ensure the consistency of data in the two sets of redis clusters.
Specifically, when a storage instruction for data to be stored is received, a first write-in task for writing the data to be stored into a first redis cluster and a second write-in task for writing the data to be stored into a second redis cluster are generated, the first write-in task is allocated to a message queue of the first redis cluster, the second write-in task is allocated to a message queue of the second redis cluster, and the storage of the data to be stored is completed after the first write-in task and the second write-in task are both successfully executed.
Step S102: and judging whether the first redis cluster and the second redis cluster are in a business peak period or not. When the first and second redis clusters are in a traffic low peak, step S103 is performed.
Specifically, according to the functional characteristics of different application systems, there are peak-to-peak traffic periods and low-to-peak traffic periods corresponding to the different application systems, for example, chat software such as QQ, wechat software, etc., the user visit volume during the period from 7 a.m. to 12 a.m. is much higher than the visit volume from 0 a.m. to 7 a.m. belongs to the peak-to-peak traffic period, and from 0 a.m. to 7 a.m. belongs to the low-to-peak traffic period. It should be understood that, the redis clusters are generally used for storing mass data, and therefore, when monitoring data consistency between the redis clusters, the amount of data that needs to be processed is huge, and more resources are occupied, and therefore, in this embodiment, in order to avoid that the comparison process of the data consistency affects normal use of the system in the peak period of service, it is preferable to perform data consistency check in the peak period of service.
Further, step S102 specifically includes:
1. and acquiring the access logs of the first redis cluster and the second redis cluster, and inquiring the access amount of the first redis cluster and the second redis cluster based on the access logs.
The access amount is an access amount per unit time, and is set in advance, for example, two hours, one hour, half an hour, or the like.
2. And judging whether the access amount is lower than a preset access amount threshold value or not.
The preset access amount threshold is preset according to the actual usage of the application.
3. And if so, the first redis cluster and the second redis cluster are in a low service peak period.
Specifically, in this embodiment, whether the first redis cluster and the second redis cluster are in a business low peak period may be determined by access amounts of the first redis cluster and the second redis cluster in a unit time, and the access amounts may be obtained through access logs of the first redis cluster and the second redis cluster. It should be understood that, in order to ensure that the access volume can be obtained, the access volume needs to be recorded in the recorded access logs of the first redis cluster and the second redis cluster, and this function may be set by a developer in advance.
Step S103: first cache data is derived from the first redis cluster and second cache data is derived from the second redis cluster.
Specifically, when the first redis cluster and the second redis cluster are in a business peak period, the first cache data is derived from the first redis cluster, and the second cache data is derived from the second redis cluster. When data are exported from the redis cluster, the rdb data file is exported from the cache of the redis cluster, a go-redis-paser tool is used for analyzing the exported rdb data file into a csv file format, keywords are classified and ordered according to data types by traversing the csv text file, and different files are generated on the basis of the data types.
Further, step S103 specifically includes:
1. and judging whether the first redis cluster and the second redis cluster are in an active mode or a cluster mode.
It should be noted that the operating modes of the redis cluster include a master-slave mode, a sentinel mode, and a cluster mode. In this embodiment, the redis cluster preferably adopts a master-slave mode or a cluster mode, and for different working modes, the data ranges to be monitored are different, and for the redis cluster in the master-slave mode, the data of each node is consistent, so that the consistency of the data of one node only needs to be monitored. For the cluster mode, the distributed storage of the redis is realized, that is, different contents are stored on each redis node, so that data on each node needs to be monitored.
2. And if the master-slave mode exists, exporting the first cache data from any node in the first redis cluster, and exporting the second cache data from any node in the second redis cluster.
3. If the cluster module is available, exporting first data of all nodes in the first redis cluster, classifying the first data according to data types to obtain first cache data, exporting second data of all nodes in the second redis cluster, and classifying the second data according to data types to obtain second cache data.
In this embodiment, the range of the data to be monitored is determined by judging the working mode of the redis cluster, so that the data of each node is prevented from being monitored by consuming a large amount of system resources in the master-slave mode.
Step S104: and when the second cache data has data inconsistent with the first cache data, sending alarm information.
Specifically, after the first cache data and the second cache data are obtained, the first cache data and the second cache data are compared one by one to confirm whether data different from the first cache data exist in the second cache data or not, and if so, alarm information is sent to remind a worker of timely maintaining the data.
Further, after sending the alarm information, the method further includes:
1. and querying the data type and the keyword of the inconsistent data in the first cache data and the second cache data.
2. And synchronizing data increments corresponding to the data types and the keywords in the first redis cluster to a second redis cluster.
Specifically, after the first cache data and the second cache data are confirmed to have different data, a keyword and a data type corresponding to the data are confirmed, corresponding data are inquired in the first redis cluster according to the data type and the keyword, and the data are incrementally synchronized to the second redis cluster, so that the data consistency of the first redis cluster and the second redis cluster is ensured.
After the data double-write monitoring method according to the first embodiment of the present invention respectively executes the write tasks of writing the data to be stored into the first redis cluster and the second redis cluster, when the system is in the service low peak, the first cache data and the second cache data are respectively derived from the first redis cluster and the second redis cluster, and then the data consistency comparison is performed on the first cache data and the second cache data, and an alarm is issued when the data are inconsistent, so as to remind a worker to maintain the data in time.
Fig. 2 is a schematic flow chart of a data double-write monitoring method according to a second embodiment of the present invention. It should be noted that the method of the present invention is not limited to the flow sequence shown in fig. 2 if the results are substantially the same. As shown in fig. 2, the method comprises the steps of:
step S201: and when a data storage instruction is received, executing a writing task of respectively writing the data to be stored into the first redis cluster and the second redis cluster.
In this embodiment, step S201 in fig. 2 is similar to step S101 in fig. 1, and for brevity, is not repeated herein.
Step S202: and judging whether the first redis cluster and the second redis cluster are in a business peak period or not. When the first and second redis clusters are not in a traffic trough, step S203 is performed.
In this embodiment, step S202 in fig. 2 is similar to step S102 in fig. 1, and for brevity, is not described herein again.
Step S203: and inquiring an operation log record of the first redis cluster, and analyzing the operation log record to obtain a keyword corresponding to the operated data.
Specifically, when the first redis cluster and the second redis cluster are not in the service peak period, in order to ensure normal operation of the system and avoid occupying too much system resources, this stage is not suitable for adopting a mode of exporting all data and then comparing to confirm whether the data are consistent. Therefore, in this embodiment, when the first redis cluster and the second redis cluster are not in the service low peak period, the operation log record of the first redis cluster is queried, and then the keyword corresponding to the operated data is obtained through analysis from the operation log record, where the operated data is data written in the first redis cluster, so as to obtain the keyword corresponding to the data written in the first redis cluster.
Step S204: and respectively acquiring first data corresponding to the keywords in the first redis cluster and second data corresponding to the keywords in the second redis cluster.
Specifically, after the keyword is obtained, corresponding first data and second data are respectively obtained from the first redis cluster and the second redis cluster.
Step S205: and when the first data is inconsistent with the second data, sending out alarm information of inconsistent data corresponding to the keyword.
Specifically, the first data and the second data are compared to determine whether the first data and the second data are consistent, if the first data and the second data are inconsistent, it is indicated that data corresponding to the keyword in the first redis cluster is modified, and data corresponding to the keyword in the second redis cluster is not modified, at this time, alarm information can be sent to modify the data corresponding to the keyword in the second redis cluster, so as to ensure data consistency. Further, when the keyword exists in the first redis cluster but does not exist in the second redis cluster, it indicates that the data corresponding to the keyword and the keyword is data newly written in the first redis cluster, and therefore, a corresponding keyword needs to be established in the second redis cluster, and then the data is written in.
Further, it should be understood that there is a time difference between writing of data in the first redis cluster and writing of data in the second redis cluster, and therefore, before sending out the alarm information that the data corresponding to the keyword is inconsistent, the method further includes:
1. and when the first data is inconsistent with the second data, recording the writing time of writing the first data corresponding to the query keyword into the first redis cluster based on the operation log.
2. And judging whether the time interval between the writing time and the current time reaches a preset time interval or not.
Specifically, the preset time interval is preset, which is generally longer than the time that would normally be consumed to write data to the second redis cluster.
3. If yes, executing alarm information of inconsistent data corresponding to the sent keyword.
Specifically, when the time interval between the write time and the current time reaches the preset time interval, it may be considered that the data write task of the second redis cluster may have failed, for example, the data write task cannot be executed due to thread blocking, and at this time, an alarm message may be sent to remind that the data corresponding to the keyword is inconsistent.
4. And if not, inquiring whether a target writing task corresponding to the keyword exists in the data writing tasks of the second redis cluster, and if the target writing task corresponding to the keyword does not exist, executing alarm information for sending inconsistent data corresponding to the keyword.
Specifically, when the time interval between the writing time and the current time does not reach the preset time interval, whether a task corresponding to the keyword exists in the data writing task of the second redis cluster is queried, if yes, the task can be waited to be executed to write the data into the second redis cluster, and if not, the alarm information that the data corresponding to the keyword is inconsistent is sent.
On the basis of the first embodiment, when the first redis cluster and the second redis cluster are in the service peak period, the method for monitoring data double writing of the second embodiment of the invention queries the operation log record of the first redis cluster to confirm the keyword of the data change in the first redis cluster, compares the keyword with the data corresponding to the corresponding keyword in the second redis cluster to confirm whether the data in the second redis cluster is successfully written in, sends corresponding warning information to the data which is not successfully written in, and reminds a worker to maintain in time, so that the consistency of the data is ensured, and the worker does not need to derive all the data for comparison, therefore, the occupied system resources are less, and the normal work of the system in the service peak period is not influenced.
Fig. 3 is a flowchart illustrating a data double-write monitoring method according to a third embodiment of the present invention. It should be noted that the method of the present invention is not limited to the flow sequence shown in fig. 3 if the results are substantially the same. As shown in fig. 3, the method comprises the steps of:
step S301: and when a data storage instruction is received, executing a writing task of respectively writing the data to be stored into the first redis cluster and the second redis cluster.
In this embodiment, step S301 in fig. 3 is similar to step S101 in fig. 1, and for brevity, is not described herein again.
Step S302: and judging whether the first redis cluster and the second redis cluster are in a business peak period or not. When the first and second redis clusters are not in a traffic trough, step S303 is performed.
In this embodiment, step S302 in fig. 3 is similar to step S102 in fig. 1, and for brevity, is not described herein again.
Step S303: and acquiring a first operation log record of the first redis cluster every a first preset time period, and analyzing to obtain a keyword list of data successfully written into the first redis cluster in the first preset time period.
Specifically, the first preset time period is preset. Every interval of a first preset time period, obtaining and analyzing a first operation log record of a first redis cluster, so as to obtain all keywords corresponding to data successfully written into the first redis cluster in the first preset time period, and constructing a keyword list by using the keywords.
Step S304: and querying a second operation log record of the second redis cluster based on the keyword list, and determining whether data corresponding to each keyword in the keyword list is successfully written into the second redis cluster within a second preset time period. If the writing is successful, go to step S305; if the writing is not successful, S306 is executed.
It should be understood that, in this embodiment, information of the keyword successfully written into the redis cluster is recorded in the operation log record of the redis cluster, so that it can be quickly determined from the operation log record which data corresponding to the keyword is successfully written into the redis cluster. Specifically, after the keyword list is obtained, a second operation log record of the second redis cluster is queried based on the keyword list, so as to confirm whether data corresponding to each keyword in the keyword list has been successfully written into the second redis cluster.
Step S305: and removing the key words corresponding to the data successfully written into the second redis cluster from the key word list.
Specifically, when data corresponding to a keyword in the keyword list is successfully written into the second redis cluster, the keyword is deleted from the keyword list.
Step S306: and marking keywords corresponding to the data which are not successfully written into the second redis cluster in the keyword list, and sending alarm information.
Specifically, when data corresponding to a keyword in the keyword list is not successfully written into the second redis cluster, the keyword is labeled in the keyword list, for example, the keyword is displayed in a conspicuous color, so that a worker can conveniently and quickly determine which keywords are not successfully written into the second redis cluster.
The data double-write monitoring method of the third embodiment of the present invention, on the basis of the first embodiment, obtains a keyword list of data successfully written into the first redis cluster from the first operation log record of the first redis cluster every a first preset time period, and then queries the second operation log record of the second redis cluster based on the keyword list to determine whether the data corresponding to the keyword in the keyword list is successfully written into the second redis cluster, if the data is successfully written into the second redis cluster, the keyword is deleted from the keyword list, and if the data is not successfully written into the second redis cluster, the keyword in the keyword list needs to be labeled to provide that the data corresponding to the keyword is not successfully written into the second redis cluster by a worker, and it also does not need to derive all data for comparison, so that the data double-write monitoring method occupies fewer system resources, and does not affect the normal operation of the system during the business peak period.
Fig. 4 is a functional block diagram of a data double-write monitoring apparatus according to an embodiment of the present invention. As shown in fig. 4, the data double-write monitoring apparatus 400 includes a storage module 401, a determination module 402, a derivation module 403, and a first alarm module 404.
The storage module 401 is configured to, when receiving a data storage instruction, execute a write task of writing data to be stored into a first redis cluster and a second redis cluster, respectively;
a determining module 402, configured to determine whether the first redis cluster and the second redis cluster are in a business peak period;
an export module 403, configured to export, when the first redis cluster and the second redis cluster are in a service peak period, first cache data from the first redis cluster and second cache data from the second redis cluster;
the first alarm module 404 is configured to send an alarm message when data inconsistent with data in the first cache data exists in the second cache data.
Optionally, the determining module 402 performs an operation of determining whether the first redis cluster and the second redis cluster are in a service peak period, where the operation specifically includes: acquiring access logs of a first redis cluster and a second redis cluster, and inquiring the access amount of the first redis cluster and the second redis cluster based on the access logs; judging whether the access amount is lower than a preset access amount threshold value or not; and if so, the first redis cluster and the second redis cluster are in a low traffic peak period.
Optionally, after the first alarm module 404 performs the operation of sending out the alarm information, it is further configured to: querying the data type and the key word of inconsistent data in the first cache data and the second cache data; and synchronizing data increments corresponding to the data types and the keywords in the first redis cluster to a second redis cluster.
Optionally, the deriving module 403 executes an operation of deriving the first cache data from the first redis cluster and deriving the second cache data from the second redis cluster, which specifically includes: judging whether the first redis cluster and the second redis cluster adopt an active mode or a cluster mode; if the master-slave mode exists, first cache data are exported from any node in the first redis cluster, and second cache data are exported from any node in the second redis cluster; and if the cluster module is the cluster module, exporting the first data of all nodes in the first redis cluster, classifying the first data according to the data type to obtain first cache data, exporting the second data of all nodes in the second redis cluster, and classifying the second data according to the data type to obtain second cache data.
Optionally, in some embodiments, as shown in fig. 5, the data double-write monitoring apparatus 400 further includes a query module 405, an obtaining module 406, and a second alarm module 407;
the query module 405 is configured to query an operation log record of the first redis cluster when the first redis cluster and the second redis cluster are not in a service low peak period, and analyze the operation log record to obtain a keyword corresponding to operated data;
an obtaining module 406, configured to obtain first data corresponding to a keyword in a first redis cluster and second data corresponding to a keyword in a second redis cluster, respectively;
and the second warning module 407 is configured to send warning information that the data corresponding to the keyword is inconsistent when the first data is inconsistent with the second data.
Optionally, before the second warning module 407 performs an operation of issuing warning information with inconsistent data corresponding to the keyword, the second warning module is further configured to: when the first data is inconsistent with the second data, recording the writing time of the first data corresponding to the query keyword written into the first redis cluster based on the operation log; judging whether the time interval between the writing time and the current time reaches a preset time interval or not; if yes, executing alarm information of inconsistent data corresponding to the sent key words; and if not, inquiring whether a target writing task corresponding to the keyword exists in the data writing tasks of the second redis cluster, and if the target writing task corresponding to the keyword does not exist, executing alarm information for sending inconsistent data corresponding to the keyword.
Optionally, in some embodiments, as shown in fig. 6, the data double-write monitoring apparatus 400 further includes a parsing module 408, a confirming module 409, a removing module 410, and a third warning module 411;
the analysis module 408 is configured to, when the first redis cluster and the second redis cluster are not in a service low peak period, obtain a first operation log record of the first redis cluster every a first preset time period, and analyze the first operation log record to obtain a keyword list of data successfully written into the first redis cluster in the first preset time period;
a confirming module 409, configured to query a second operation log record of a second redis cluster based on the keyword list, and confirm whether data corresponding to each keyword in the keyword list is successfully written into the second redis cluster within a second preset time period;
a removing module 410, configured to remove, if the writing is successful, a keyword corresponding to data successfully written into the second redis cluster from the keyword list;
the third warning module 411 is configured to mark, if the writing is unsuccessful, a keyword corresponding to the data that is not successfully written into the second redis cluster in the keyword list, and send warning information.
For other details of the technical solutions implemented by the modules in the data double-write monitoring apparatus in the foregoing embodiments, reference may be made to the description of the data double-write monitoring method in the foregoing embodiments, and details are not described here again.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and reference may be made to the partial description of the method embodiment for relevant points.
Referring to fig. 7, fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in fig. 7, the computer device 70 includes a processor 71 and a memory 72 coupled to the processor 71, wherein the memory 72 stores program instructions, and the program instructions, when executed by the processor 71, cause the processor 71 to execute the steps of the data double-write monitoring method according to any of the above embodiments.
The processor 71 may also be referred to as a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip having signal processing capabilities. The processor 71 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Referring to fig. 8, fig. 8 is a schematic structural diagram of a storage medium according to an embodiment of the invention. The storage medium of the embodiment of the present invention stores a program instruction 81 capable of implementing all the methods described above, where the program instruction 81 may be stored in the storage medium in the form of a software product, and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) or a Processor (Processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a mobile hard disk, a Read-Only Memory (Rom), a Random Access Memory (Ram), a magnetic disk or an optical disk, or a computer device, such as a computer, a server, a mobile phone, or a tablet.
In the several embodiments provided in the present application, it should be understood that the disclosed computer apparatus, device and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. The above embodiments are merely examples and are not intended to limit the scope of the present disclosure, and all modifications, equivalents, and flow charts using the contents of the specification and drawings of the present disclosure or those directly or indirectly applied to other related technical fields are intended to be included in the scope of the present disclosure.

Claims (10)

1. A data double-write monitoring method is characterized by comprising the following steps:
when a data storage instruction is received, executing a write-in task of respectively writing data to be stored into a first redis cluster and a second redis cluster;
judging whether the first redis cluster and the second redis cluster are in a service peak period or not;
when the first and second redis clusters are in a business low peak period, deriving first cache data from the first redis cluster and deriving second cache data from the second redis cluster;
and when the second cache data contains data inconsistent with the first cache data, sending out alarm information.
2. The method for monitoring data double writing according to claim 1, wherein the determining whether the first redis cluster and the second redis cluster are in a traffic low peak includes:
acquiring access logs of the first redis cluster and the second redis cluster, and querying the access amount of the first redis cluster and the second redis cluster based on the access logs;
judging whether the access amount is lower than a preset access amount threshold value or not;
if so, the first redis cluster and the second redis cluster are in the traffic low peak period.
3. The data double-write monitoring method according to claim 1, further comprising, after sending the warning information:
querying the data type and the key word of inconsistent data in the first cache data and the second cache data;
and synchronizing the data type and the data increment corresponding to the keyword in the first redis cluster to the second redis cluster.
4. The method of data double-write monitoring of claim 1, wherein the deriving first cached data from the first redis cluster and second cached data from the second redis cluster comprises:
judging whether the first redis cluster and the second redis cluster are in an active mode or a cluster mode;
if the master-slave mode is adopted, the first cache data is derived from any node in the first redis cluster, and the second cache data is derived from any node in the second redis cluster;
if the cluster module is available, exporting the first data of all nodes in the first redis cluster, classifying the first data according to data types to obtain the first cache data, exporting the second data of all nodes in the second redis cluster, and classifying the second data according to data types to obtain the second cache data.
5. The method for monitoring data double writing according to claim 1, wherein the determining whether the first redis cluster and the second redis cluster are after a traffic low peak period further comprises:
when the first redis cluster and the second redis cluster are not in a business peak period, inquiring an operation log record of the first redis cluster, and analyzing the operation log record to obtain a keyword corresponding to operated data;
respectively acquiring first data corresponding to the keyword in the first redis cluster and second data corresponding to the keyword in the second redis cluster;
and when the first data is inconsistent with the second data, sending out alarm information of inconsistency of data corresponding to the keyword.
6. The method for monitoring data double writing according to claim 5, wherein before the sending out the alarm information that the data corresponding to the keyword is inconsistent, the method further comprises:
when the first data and the second data are inconsistent, inquiring the writing time of the first data corresponding to the keyword written into the first redis cluster based on the operation log record;
judging whether the time interval between the writing time and the current time reaches a preset time interval or not;
if yes, executing to send out alarm information of inconsistent data corresponding to the keyword;
if not, inquiring whether a target writing task corresponding to the keyword exists in the data writing tasks of the second redis cluster, and if the target writing task corresponding to the keyword does not exist, executing to send out alarm information of inconsistent data corresponding to the keyword.
7. The method for monitoring data double writing according to claim 1, wherein the determining whether the first redis cluster and the second redis cluster are after a traffic low peak period further comprises:
when the first redis cluster and the second redis cluster are not in a service peak period, acquiring a first operation log record of the first redis cluster at intervals of a first preset time period, and analyzing to obtain a keyword list of data successfully written into the first redis cluster in the first preset time period;
querying a second operation log record of the second redis cluster based on the keyword list, and determining whether data corresponding to each keyword in the keyword list is successfully written into the second redis cluster within a second preset time period;
if the data is successfully written in, removing the keyword corresponding to the data successfully written in the second redis cluster from the keyword list;
and if the data is not successfully written in, marking the keyword corresponding to the data which is not successfully written in the second redis cluster in the keyword list, and sending out alarm information.
8. A data double write monitoring apparatus, comprising:
the storage module is used for executing write-in tasks of respectively writing data to be stored into a first redis cluster and a second redis cluster when receiving a data storage instruction;
the judging module is used for judging whether the first redis cluster and the second redis cluster are in a service low peak period or not;
an export module, configured to export first cache data from the first redis cluster and second cache data from the second redis cluster when the first and second redis clusters are in a traffic trough;
and the first alarm module is used for sending out alarm information when the second cache data has data inconsistent with the data in the first cache data.
9. A computer device, characterized in that the computer device comprises a processor, a memory coupled to the processor, in which memory program instructions are stored which, when executed by the processor, cause the processor to carry out the steps of the data double write monitoring method according to any one of claims 1 to 7.
10. A storage medium storing program instructions capable of implementing the data double write monitoring method according to any one of claims 1 to 7.
CN202210936957.1A 2022-08-05 2022-08-05 Data double-writing monitoring method, device, equipment and storage medium Pending CN115269623A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210936957.1A CN115269623A (en) 2022-08-05 2022-08-05 Data double-writing monitoring method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210936957.1A CN115269623A (en) 2022-08-05 2022-08-05 Data double-writing monitoring method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115269623A true CN115269623A (en) 2022-11-01

Family

ID=83748727

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210936957.1A Pending CN115269623A (en) 2022-08-05 2022-08-05 Data double-writing monitoring method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115269623A (en)

Similar Documents

Publication Publication Date Title
CN111339175B (en) Data processing method, device, electronic equipment and readable storage medium
US11016994B2 (en) System for performing on-line transaction processing and on-line analytical processing on runtime data
US10296435B2 (en) Storage of mass data for monitoring
US7509539B1 (en) Method for determining correlation of synchronized event logs corresponding to abnormal program termination
US20080010497A1 (en) Selecting a Logging Method via Metadata
CN106487574A (en) Automatic operating safeguards monitoring system
JP6160064B2 (en) Application determination program, failure detection apparatus, and application determination method
TWI738721B (en) Task scheduling method and device
CN110535713B (en) Monitoring management system and monitoring management method
CN105512283A (en) Data quality management and control method and device
CN107506266B (en) Data recovery method and system
CN108228322B (en) Distributed link tracking and analyzing method, server and global scheduler
EP2723017A1 (en) Method, apparatus and system for implementing distributed auto-incrementing counting
JP2020057416A (en) Method and device for processing data blocks in distributed database
CN114154035A (en) Data processing system for dynamic loop monitoring
CN107515807B (en) Method and device for storing monitoring data
CN111339466A (en) Interface management method and device, electronic equipment and readable storage medium
CN114637656B (en) Redis-based monitoring method and device, storage medium and equipment
CN102986151A (en) Monitoring system and data transfer device and method thereof
CN115269623A (en) Data double-writing monitoring method, device, equipment and storage medium
CN114860432A (en) Method and device for determining information of memory fault
CN114896128A (en) Application program performance testing method and device based on block chain
CN112860746B (en) Cache reduction-based method, equipment and system
CN113961611A (en) Log processing method, device, equipment and storage medium
US8032797B1 (en) Storage of mass data for monitoring

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination