CN116827828A - Database master-slave delay monitoring method and device, electronic equipment and storage medium - Google Patents

Database master-slave delay monitoring method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN116827828A
CN116827828A CN202311037719.8A CN202311037719A CN116827828A CN 116827828 A CN116827828 A CN 116827828A CN 202311037719 A CN202311037719 A CN 202311037719A CN 116827828 A CN116827828 A CN 116827828A
Authority
CN
China
Prior art keywords
database
library
standby
main library
main
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311037719.8A
Other languages
Chinese (zh)
Inventor
陈德虎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Pingan Payment Technology Service Co Ltd
Original Assignee
Pingan Payment Technology Service Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Pingan Payment Technology Service Co Ltd filed Critical Pingan Payment Technology Service Co Ltd
Priority to CN202311037719.8A priority Critical patent/CN116827828A/en
Publication of CN116827828A publication Critical patent/CN116827828A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/08Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852Delays
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1469Backup restoration techniques
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing data switching networks
    • H04L43/10Active monitoring, e.g. heartbeat, ping or trace-route
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Environmental & Geological Engineering (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to the technical field of cloud service monitoring, and provides a method, a device, electronic equipment and a storage medium for monitoring main and standby delays of a database. By the method, the specific primary and standby delay time can be obtained, and a null value is not returned as in the prior art. In addition, the calculated main and standby delay time is higher in accuracy through the current system time of the standby library and the current system time of the main library, and the cliff-breaking decline can not occur. Furthermore, the calculated active-standby delay time can be accurate to microseconds.

Description

Database master-slave delay monitoring method and device, electronic equipment and storage medium
Technical Field
The invention relates to the technical field of cloud service monitoring, in particular to a database main and standby delay monitoring method, a device, electronic equipment and a storage medium.
Background
The database is composed of a host and a standby machine in a host-standby replication environment, and the standby machine needs to synchronize the host log and replay the host log. The synchronous host log refers to that a receiving thread of the standby machine receives the host log from the network and persists the host log into a local file, and the playback host log refers to that the playback thread reads the content of the synchronous log and executes the operation content recorded in the log on the standby machine.
Due to network bandwidth, distance between the master node and the slave node, etc., delay is generated in synchronization of the slave and playback of the log. Accurately counting the synchronization delay and playback delay is particularly important in practical applications. The synchronization delay determines the amount of data lost after the disaster of the host, and the larger the synchronization delay is, the more data is lost after the disaster of the host. In an actual system, the synchronization delay needs to be monitored in real time, if the synchronization delay is found to exceed the threshold value, an active alarm is needed, and corresponding measures are executed to reduce the synchronization delay. The playback delay is related to the visibility of the data when the standby is a read device, and if the playback delay is too great, the user cannot read the already committed transaction on the standby. In addition, playback delay is also related to the time it takes for a host to fail-over after a disaster. If the playback delay is too great, the failover time will be long, resulting in a system that cannot serve outside for a long period of time. Similarly, real-time monitoring of playback delay is required in an actual system, and if the playback delay is found to exceed the threshold value, an active alarm is required, and corresponding measures are implemented to reduce the synchronization delay.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a method, an apparatus, an electronic device, and a storage medium for monitoring a database primary-backup delay, which can improve accuracy of database primary-backup delay monitoring.
The first aspect of the invention provides a method for monitoring the primary and backup delays of a database, which comprises the following steps:
synchronizing the first heartbeat table of the database main library into the database standby library to obtain a second heartbeat table;
starting a first thread in the database standby library, and acquiring the current system time of the database standby library through a preset time period of the first thread Cheng Meige;
acquiring a target time corresponding to a target field in the second heartbeat table through the preset time period of the first line Cheng Meige, wherein the target time is the current system time of the database main library, and the time format of the target field is accurate to microseconds;
and obtaining the main and standby delay time of the database according to the current system time and the target time.
According to an alternative embodiment of the present invention, before the synchronizing the first heartbeat table of the database primary repository into the database backup repository, the method further includes:
Creating the first heartbeat table in the database main library;
starting a second thread in the database main library, and writing the current system time of the database main library into the target field in the first heartbeat table through the second thread;
updating the current system time of the database main library every other preset time period through the second thread;
and synchronizing the updated current system time in the database main library to the target field of the database standby library.
According to an alternative embodiment of the present invention, updating, by the second thread, the current system time of the database master library every the preset time period includes:
and updating the current system time of the database main library in the memory at intervals of the preset time period through the second thread.
According to an alternative embodiment of the invention, the method further comprises:
acquiring process state information and health state information of the database main library;
judging whether the database main library has faults or not according to the process state information and the health state information;
and if the database main library is determined to have faults, controlling the database main library and the database standby library to perform main-standby switching operation.
According to an optional embodiment of the invention, the determining whether the database primary library malfunctions according to the process state information and the health state information includes:
if the process of the database main library is determined to be absent, and the read health state information is information representing that the database main library is in an unhealthy state currently, determining that the database main library fails;
and if the process of the database main library is determined to exist, or the read health state information is information representing that the database main library is in a health state currently, determining that the database main library has no fault.
According to an alternative embodiment of the invention, the method further comprises:
generating a database log;
storing user operations on the database master library in the database log;
calculating the total times of the user operation in a preset time period and the times of target operation on each column in the database main library;
generating an index column name according to the total times of the user operation and the times of target operation on each column in the database main library;
Creating an index of the database main library according to the index column name in a preset period;
the preset time period is a time period with the lowest database access amount in the preset time period.
According to an optional embodiment of the present invention, the generating an index column name according to the total number of the user operations and the number of target operations performed on each column in the database main library includes:
calculating to obtain the number of times of target operation according to the total number of times of user operation and the number of times of target operation on each column in the database main library;
judging whether the frequency duty ratio is higher than a preset duty ratio threshold value or not;
when the frequency occupation ratio is higher than the preset occupation ratio threshold value, creating an index column name for a column corresponding to the target operation in the database main library;
the remaining columns in the database master library do not create index column names.
A second aspect of the present invention provides a database active/standby delay monitoring apparatus, the apparatus comprising:
the heartbeat synchronization module is used for synchronizing the first heartbeat table of the database main library into the database standby library to obtain a second heartbeat table;
the first obtaining module is used for starting a first thread in the database standby library and obtaining the current system time of the database standby library through the first thread Cheng Meige in a preset time period;
The second obtaining module is configured to obtain, according to the preset time period of the first line Cheng Meige, a target time corresponding to a target field in the second heartbeat table, where the target time is a current system time of the database main library, and a time format of the target field is accurate to microseconds;
and the delay monitoring module is used for obtaining the main and standby delay time of the database according to the current system time and the target time.
A third aspect of the present invention provides an electronic device, the electronic device comprising a processor and a memory, the processor being configured to implement the database master-slave delay monitoring method when executing a computer program stored in the memory.
A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the database master-slave latency monitoring method.
According to the method, the device, the electronic equipment and the storage medium for monitoring the database main and standby delays, the first heartbeat table of the database main and standby database is synchronized to the database standby database to obtain the second heartbeat table, the first thread in the database standby database is started, the current system time of the database standby database is obtained through the first thread Cheng Meige in a preset time period, meanwhile, the target time corresponding to the target field in the second heartbeat table is obtained, the target time is the current system time of the database main and standby delays, and the main and standby delays of the database are obtained according to the current system time and the target time. Regardless of whether the replication thread of the database standby library is disconnected, the method of the invention can obtain the main standby delay time of the specific database instead of returning a null value as in the prior art. In addition, the calculated main and standby delay time is higher in accuracy through the current system time of the database standby library and the current system time of the database main library, and the cliff-type decline can not occur. Furthermore, since the time format of the target field can be accurate to microseconds, the calculated master-slave delay time of the database can also be accurate to microseconds.
Drawings
Fig. 1 is a flowchart of a method for monitoring a database primary and backup delay according to an embodiment of the present invention.
Fig. 2 is a block diagram of a database master-slave delay monitoring device according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will be more clearly understood, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Embodiments of the invention and features of the embodiments may be combined with each other without conflict.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing embodiments of the invention only and is not intended to be limiting of the invention.
The database main and standby delay monitoring method provided by the embodiment of the invention is executed by the electronic equipment, and correspondingly, the database main and standby delay monitoring device is operated in the electronic equipment.
The embodiment of the invention can perform standardized processing on the data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.
Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.
Example 1
Fig. 1 is a flowchart of a method for monitoring a database primary and backup delay according to an embodiment of the present invention. The method for monitoring the database master-slave delay specifically comprises the following steps, the sequence of the steps in the flow chart can be changed according to different requirements, and some steps can be omitted.
S11, synchronizing the first heartbeat table of the database main library into the database standby library to obtain a second heartbeat table.
The primary library and the standby library in this embodiment may form a database cluster, where the primary library and the standby library may be understood as 2 database nodes in the database cluster, and the whole cluster serves as a server to provide services to the outside.
In the prior art, the second_Behind_Master value (in Seconds) is typically obtained by executing show slave status command on the standby library, which represents the time the current standby library is delayed from the primary library, but this state variable reflects the copy delay in Seconds and cannot achieve more accurate delay detection. Further, the default second_Behind_Master value is obtained by comparing the current timestamp of the server with the timestamp of the event in the binary log, so that the delay can be reported only when the event is executed, that is, when the event is not executed. That is, the standby library copy thread is not running, the second_Behind_Master value is null, it is not known how long the delay is, and only the copy is broken. In addition, when a large transaction is executed, since update data of a large transaction is submitted up to one hour, and the update is recorded in the binary log one hour later than the actual occurrence time of the transaction, when the standby library runs the statement, the standby library delay time is temporarily reported as one hour, and becomes 0 soon after the execution is finished, which is quite abrupt.
It can be seen that the prior art has the following defects:
the second_Behind_Master value indicates inaccurate time, but only a rough result and has a large error;
the accuracy of the second_bearing_master value can only reach the second level, and can not meet the service scene with higher accuracy of the main and standby delay time, such as financial service.
In the embodiment of the invention, the first heartbeat table is firstly created in the database main library, and the first heartbeat table in the database main library is synchronized to the database standby library after the first heartbeat table is created in the database main library due to the synchronization mechanism between the database main library and the database standby library, so that the second heartbeat table in the database standby library is obtained.
S12, starting a first thread in the database standby library, and acquiring the current system time of the database standby library through the first thread Cheng Meige for a preset time period.
Wherein the preset time period may be set to 1 second(s).
And starting a first thread in the database standby library, and acquiring the current system time of the database standby library every 1 s.
The current system time format of the database backup library, such as 2019-08-18T16:48:53.003130, can be accurate to microseconds, and can meet 99.99% of use scenes.
S13, acquiring target time corresponding to a target field in the second heartbeat table through the first line Cheng Meige in the preset time period.
Wherein the target field may be a ts field. And the target time corresponding to the target field is the current system time of the database main library.
It should be appreciated that, since the second heartbeat table is obtained synchronously from the first heartbeat table in the database main library, the target field in the second heartbeat table is consistent with the target field in the first heartbeat table, and the time of recording the target field in the second heartbeat table is the current system time of the database main library stored by the target field in the first heartbeat table.
S14, obtaining the main and standby delay time of the database according to the current system time and the target time.
And subtracting the target time from the current system time to obtain a time difference which is the main and standby delay time.
In an alternative embodiment, before the synchronizing the first heartbeat table of the database primary repository into the database backup repository, the method further includes:
creating the first heartbeat table in the database main library;
starting a second thread in the database main library, and writing the current system time of the database main library into the target field in the first heartbeat table through the second thread;
Updating the current system time of the database main library every other preset time period through the second thread;
and synchronizing the updated current system time in the database main library to the target field of the database standby library.
First, a first heartbeat table is created in a database main library, the first heartbeat table is named as a heartbeat table, and the table structure of the first heartbeat table is as follows:
CREATE TABLE heartbeat(
ts varchar(26)NOT NULL,
server_id int unsigned NOT NULL PRIMARY KEY,
file varchar(255)DEFAULT NULL,
position bigint unsigned DEFAULT NULL,
relay_master_log_file varchar(255)DEFAULT NULL,
exec_master_log_posbigint unsigned DEFAULT NULL
)。
in one embodiment, a target field may be set in the first heartbeat table to store the current system time of the database master library. The target field may be a ts field.
And starting a second thread in the database main library, and updating the current system time of the database main library by the second thread every preset time period. Specifically, the current system time of the database main library is acquired every preset time period, and the acquired current system time is covered by the stored time in the target field. The target field set in the first heartbeat table and the updated current system time in the target field in the first heartbeat table are synchronized into a database standby library.
In an optional embodiment, the updating, by the second thread, the current system time of the database master library every the preset time period includes: and updating the current system time of the database main library in the memory at intervals of the preset time period through the second thread.
In the above embodiment, the performance of the database main library is not affected by updating the current system time of the database main library in the memory.
According to the embodiment of the invention, the first heartbeat table of the database main library is synchronized to the database standby library to obtain the second heartbeat table, the first thread in the database standby library is started, the current system time of the database standby library is obtained through the first thread Cheng Meige in a preset time period, meanwhile, the target time corresponding to the target field in the second heartbeat table is obtained, the target time is the current system time of the database main library, and the main and standby delay time of the database is obtained according to the current system time and the target time. Regardless of whether the replication thread of the database standby library is disconnected, the method of the invention can obtain the main standby delay time of the specific database instead of returning a null value as in the prior art. In addition, the calculated main and standby delay time is higher in accuracy through the current system time of the database standby library and the current system time of the database main library, and the cliff-type decline can not occur. Furthermore, since the time format of the target field can be accurate to microseconds, the calculated master-slave delay time of the database can also be accurate to microseconds.
In an alternative embodiment, the method further comprises:
acquiring process state information and health state information of the database main library;
judging whether the database main library has faults or not according to the process state information and the health state information;
and if the database main library is determined to have faults, controlling the database main library and the database standby library to perform main-standby switching operation.
In the above embodiment, the regular and periodic execution process judgment instruction can be performed through the watchdog component. For example, the process determination instruction is executed every 10 minutes: "ps-ef|greppastgres|grep database", to determine whether a process of the database master library exists. If the execution result is that the process list is not empty, the process of the database main library is not lost, namely the process of the database main library exists. If the execution result is that the process list is empty, the process of the database main library is lost, namely, the process of the database main library does not exist. Wherein, "ps-ef|greppastgres|grep database" is a prior art, and the present invention is not described herein.
In the above embodiment, the health status information recorded in the instance status table for the database main library may be read at regular time. Optionally, pgpool can monitor the instance running condition of the database master library and provide a corresponding instance state table pool_nodes. The invention can periodically read the value of the state in the example state table pool_nodes. If the state is "up", the state indicates that the health state of the database main library is "healthy", and if the state is "down", the state indicates that the health state of the database main library is "unhealthy".
After determining that the database main library fails, indicating that the database main library does not have the condition of serving as the main database to provide service to the outside, the main-standby switching operation needs to be performed in time, so that the database main library is switched to the database standby library, and the failure recovery is performed.
In an optional embodiment, the determining whether the database primary library fails according to the process state information and the health state information includes:
if the process of the database main library is determined to be absent, and the read health state information is information representing that the database main library is in an unhealthy state currently, determining that the database main library fails;
and if the process of the database main library is determined to exist, or the read health state information is information representing that the database main library is in a health state currently, determining that the database main library has no fault.
If the result of executing "ps-ef|grepostgres|grepdatabase" is that the process is lost (i.e., the process does not exist) and the value of the state in the read pool_nodes is "down", then it is determined that the database master library fails.
The failure of the database main library comprises the following 3 cases:
the process of the database main library exists, and the read health state information is information representing that the database main library is in a health state currently;
the process of the database main library does not exist, but the read health state information is information representing that the database main library is in a health state currently;
the process of the database main library exists, but the read health state information is information representing that the database main library is in an unhealthy state currently.
Currently, indexes of the database are manually created according to actual use conditions of the system or daily experience, and are optimized. However, because the accessing person is changing from time to time, the artificially created static index may only be able to deal with a certain period of scenes, or some scenes in the past. While other periods outside of a particular period, or future changes, the previously created static index is often not applicable.
In an alternative embodiment, the method further comprises:
generating a database log;
Storing user operations on the database master library in the database log;
calculating the total times of the user operation in a preset time period and the times of target operation on each column in the database main library;
generating an index column name according to the total times of the user operation and the times of target operation on each column in the database main library;
and creating an index of the database main library according to the index column name in a preset period.
Wherein, the user operation on the database main library can include, but is not limited to: query, insert, delete, modify, add, copy, etc.
The target operation performed on each column in the database main library is a query.
Creating a database log, recording database operation on each item of content in the database, calculating the number of times of inquiry to each column of the database main library in a preset time period, thereby calculating the number of times of inquiry to each column, and because the inquiry speed can be improved by using the index during inquiry, the index is required to be modified by using the index during insertion, modification and the like, the performance is reduced, so that whether the column needs to be created or not can be judged according to the number of times of inquiry, an index column name is generated according to the calculated number of times of inquiry, the database index is created according to the index column name in the preset time period, automatic creation of the optimal index to the database is realized, and the operation is updated.
In an optional implementation manner, the generating the index column name according to the total number of the user operations and the number of target operations performed on each column in the database main library includes:
calculating to obtain the number of times of target operation according to the total number of times of user operation and the number of times of target operation on each column in the database main library;
judging whether the frequency duty ratio is higher than a preset duty ratio threshold value or not;
when the frequency occupation ratio is higher than the preset occupation ratio threshold value, creating an index column name for a column corresponding to the target operation in the database main library;
the remaining columns in the database master library do not create index column names.
Wherein the preset duty cycle threshold may be set to 50%.
And creating an index column name for the column with high query times, and not creating the index column name for the column with high insertion or modification times, thereby improving the query speed and the database performance.
In an alternative embodiment, the preset period is a period with the lowest database access amount in the preset period. In the period of lowest access, the index can be updated in time without affecting the access of the user to the database.
Example two
Fig. 2 is a block diagram of a database master-slave delay monitoring device according to a second embodiment of the present invention.
In some embodiments, the database master delay monitor 20 may include a plurality of functional modules comprised of computer program segments. The computer program of each program segment in the database primary and secondary delay monitor apparatus 20 may be stored in a memory of an electronic device and executed by at least one processor to perform the functions of database primary and secondary delay monitoring (described in detail with reference to fig. 1).
In this embodiment, the database master/slave delay monitor apparatus 20 may be divided into a plurality of functional modules according to the functions performed by the database master/slave delay monitor apparatus. The functional module may include: the system comprises a heartbeat synchronization module 201, a first acquisition module 202, a second acquisition module 203, a delay monitoring module 204, a master-slave switching module 205 and an index creation module 206. The module referred to in the present invention refers to a series of computer program segments capable of being executed by at least one processor and of performing a fixed function, stored in a memory. In the present embodiment, the functions of the respective modules will be described in detail in the following embodiments.
The heartbeat synchronization module 201 is configured to synchronize a first heartbeat table of the database primary library to the database standby library to obtain a second heartbeat table.
The primary library and the standby library in this embodiment may form a database cluster, where the primary library and the standby library may be understood as 2 database nodes in the database cluster, and the whole cluster serves as a server to provide services to the outside.
In the prior art, the second_Behind_Master value (in Seconds) is typically obtained by executing show slave status command on the standby library, which represents the time the current standby library is delayed from the primary library, but this state variable reflects the copy delay in Seconds and cannot achieve more accurate delay detection. Further, the default second_Behind_Master value is obtained by comparing the current timestamp of the server with the timestamp of the event in the binary log, so that the delay can be reported only when the event is executed, that is, when the event is not executed. That is, the standby library copy thread is not running, the second_Behind_Master value is null, it is not known how long the delay is, and only the copy is broken. In addition, when a large transaction is executed, since update data of a large transaction is submitted up to one hour, and the update is recorded in the binary log one hour later than the actual occurrence time of the transaction, when the standby library runs the statement, the standby library delay time is temporarily reported as one hour, and becomes 0 soon after the execution is finished, which is quite abrupt.
It can be seen that the prior art has the following defects:
the second_Behind_Master value indicates inaccurate time, but only a rough result and has a large error;
the accuracy of the second_bearing_master value can only reach the second level, and can not meet the service scene with higher accuracy of the main and standby delay time, such as financial service.
In the embodiment of the invention, the first heartbeat table is firstly created in the database main library, and the first heartbeat table in the database main library is synchronized to the database standby library after the first heartbeat table is created in the database main library due to the synchronization mechanism between the database main library and the database standby library, so that the second heartbeat table in the database standby library is obtained.
The first obtaining module 202 is configured to start a first thread in the database standby library, and obtain a current system time of the database standby library through the first thread Cheng Meige for a preset period of time.
Wherein the preset time period may be set to 1 second(s).
And starting a first thread in the database standby library, and acquiring the current system time of the database standby library every 1 s.
The current system time format of the database backup library, such as 2019-08-18T16:48:53.003130, can be accurate to microseconds, and can meet 99.99% of use scenes.
The second obtaining module 203 is configured to obtain, according to the preset time period of the first line Cheng Meige, a target time corresponding to a target field in the second heartbeat table.
Wherein the target field may be a ts field. And the target time corresponding to the target field is the current system time of the database main library.
It should be appreciated that, since the second heartbeat table is obtained synchronously from the first heartbeat table in the database main library, the target field in the second heartbeat table is consistent with the target field in the first heartbeat table, and the time of recording the target field in the second heartbeat table is the current system time of the database main library stored by the target field in the first heartbeat table.
The delay monitoring module 204 is configured to obtain a master-slave delay time of the database according to the current system time and the target time.
And subtracting the target time from the current system time to obtain a time difference which is the main and standby delay time.
In an alternative embodiment, the heartbeat synchronization module 201 is further configured to, before the synchronizing the first heartbeat table of the database primary library into the database backup library:
creating the first heartbeat table in the database main library;
Starting a second thread in the database main library, and writing the current system time of the database main library into the target field in the first heartbeat table through the second thread;
updating the current system time of the database main library every other preset time period through the second thread;
and synchronizing the updated current system time in the database main library to the target field of the database standby library.
First, a first heartbeat table is created in a database main library, the first heartbeat table is named as a heartbeat table, and the table structure of the first heartbeat table is as follows:
CREATE TABLE heartbeat(
tsvarchar(26)NOT NULL,
server_id int unsigned NOT NULL PRIMARY KEY,
file varchar(255)DEFAULT NULL,
position bigint unsigned DEFAULT NULL,
relay_master_log_filevarchar(255)DEFAULT NULL,
exec_master_log_posbigint unsigned DEFAULT NULL
)。
in one embodiment, a target field may be set in the first heartbeat table to store the current system time of the database master library. The target field may be a ts field.
And starting a second thread in the database main library, and updating the current system time of the database main library by the second thread every preset time period. Specifically, the current system time of the database main library is acquired every preset time period, and the acquired current system time is covered by the stored time in the target field. The target field set in the first heartbeat table and the updated current system time in the target field in the first heartbeat table are synchronized into a database standby library.
In an optional embodiment, the updating, by the second thread, the current system time of the database master library every the preset time period includes: and updating the current system time of the database main library in the memory at intervals of the preset time period through the second thread.
In the above embodiment, the performance of the database main library is not affected by updating the current system time of the database main library in the memory.
According to the embodiment of the invention, the first heartbeat table of the database main library is synchronized to the database standby library to obtain the second heartbeat table, the first thread in the database standby library is started, the current system time of the database standby library is obtained through the first thread Cheng Meige in a preset time period, meanwhile, the target time corresponding to the target field in the second heartbeat table is obtained, the target time is the current system time of the database main library, and the main and standby delay time of the database is obtained according to the current system time and the target time. Regardless of whether the replication thread of the database standby library is disconnected, the method of the invention can obtain the main standby delay time of the specific database instead of returning a null value as in the prior art. In addition, the calculated main and standby delay time is higher in accuracy through the current system time of the database standby library and the current system time of the database main library, and the cliff-type decline can not occur. Furthermore, since the time format of the target field can be accurate to microseconds, the calculated master-slave delay time of the database can also be accurate to microseconds.
The primary-backup switching module 205 is configured to perform primary-backup switching operation on the database primary library and the database backup library.
In an alternative embodiment, the performing a primary-backup switching operation on the database primary library and the database backup library includes:
acquiring process state information and health state information of the database main library;
judging whether the database main library has faults or not according to the process state information and the health state information;
and if the database main library is determined to have faults, controlling the database main library and the database standby library to perform main-standby switching operation.
In the above embodiment, the regular and periodic execution process judgment instruction can be performed through the watchdog component. For example, the process determination instruction is executed every 10 minutes: "ps-ef|greppastgres|grep database", to determine whether a process of the database master library exists. If the execution result is that the process list is not empty, the process of the database main library is not lost, namely the process of the database main library exists. If the execution result is that the process list is empty, the process of the database main library is lost, namely, the process of the database main library does not exist. Wherein, "ps-ef|greppastgres|grep database" is a prior art, and the present invention is not described herein.
In the above embodiment, the health status information recorded in the instance status table for the database main library may be read at regular time. Optionally, pgpool can monitor the instance running condition of the database master library and provide a corresponding instance state table pool_nodes. The invention can periodically read the value of the state in the example state table pool_nodes. If the state is "up", the state indicates that the health state of the database main library is "healthy", and if the state is "down", the state indicates that the health state of the database main library is "unhealthy".
After determining that the database main library fails, indicating that the database main library does not have the condition of serving as the main database to provide service to the outside, the main-standby switching operation needs to be performed in time, so that the database main library is switched to the database standby library, and the failure recovery is performed.
In an optional embodiment, the determining whether the database primary library fails according to the process state information and the health state information includes:
If the process of the database main library is determined to be absent, and the read health state information is information representing that the database main library is in an unhealthy state currently, determining that the database main library fails;
and if the process of the database main library is determined to exist, or the read health state information is information representing that the database main library is in a health state currently, determining that the database main library has no fault.
If the result of executing "ps-ef|grepostgres|grep database" is that the process is lost (i.e., the process does not exist) and the value of the state in the read pool_nodes is "down", then it is determined that the database master library has failed.
The failure of the database main library comprises the following 3 cases:
the process of the database main library exists, and the read health state information is information representing that the database main library is in a health state currently;
the process of the database main library does not exist, but the read health state information is information representing that the database main library is in a health state currently;
the process of the database main library exists, but the read health state information is information representing that the database main library is in an unhealthy state currently.
Currently, indexes of the database are manually created according to actual use conditions of the system or daily experience, and are optimized. However, because the accessing person is changing from time to time, the artificially created static index may only be able to deal with a certain period of scenes, or some scenes in the past. While other periods outside of a particular period, or future changes, the previously created static index is often not applicable.
The index creation module 206 is configured to create an index for the database master library.
In an alternative embodiment, said creating an index for said database master library comprises:
generating a database log;
storing user operations on the database master library in the database log;
calculating the total times of the user operation in a preset time period and the times of target operation on each column in the database main library;
generating an index column name according to the total times of the user operation and the times of target operation on each column in the database main library;
and creating an index of the database main library according to the index column name in a preset period.
Wherein, the user operation on the database main library can include, but is not limited to: query, insert, delete, modify, add, copy, etc.
The target operation performed on each column in the database main library is a query.
Creating a database log, recording database operation on each item of content in the database, calculating the number of times of inquiry to each column of the database main library in a preset time period, thereby calculating the number of times of inquiry to each column, and because the inquiry speed can be improved by using the index during inquiry, the index is required to be modified by using the index during insertion, modification and the like, the performance is reduced, so that whether the column needs to be created or not can be judged according to the number of times of inquiry, an index column name is generated according to the calculated number of times of inquiry, the database index is created according to the index column name in the preset time period, automatic creation of the optimal index to the database is realized, and the operation is updated.
In an optional implementation manner, the generating the index column name according to the total number of the user operations and the number of target operations performed on each column in the database main library includes:
Calculating to obtain the number of times of target operation according to the total number of times of user operation and the number of times of target operation on each column in the database main library;
judging whether the frequency duty ratio is higher than a preset duty ratio threshold value or not;
when the frequency occupation ratio is higher than the preset occupation ratio threshold value, creating an index column name for a column corresponding to the target operation in the database main library;
the remaining columns in the database master library do not create index column names.
Wherein the preset duty cycle threshold may be set to 50%.
And creating an index column name for the column with high query times, and not creating the index column name for the column with high insertion or modification times, thereby improving the query speed and the database performance.
In an alternative embodiment, the preset period is a period with the lowest database access amount in the preset period. In the period of lowest access, the index can be updated in time without affecting the access of the user to the database.
Example III
The present embodiment provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements the steps in the above-described database master-slave delay monitoring method embodiment, for example, S11-S14 shown in fig. 1:
S11, synchronizing a first heartbeat table of a database main library into a database standby library to obtain a second heartbeat table;
s12, starting a first thread in the database standby library, and acquiring the current system time of the database standby library through the first thread Cheng Meige in a preset time period;
s13, acquiring target time corresponding to a target field in the second heartbeat table through the first line Cheng Meige in the preset time period;
s14, obtaining the main and standby delay time of the database according to the current system time and the target time.
Alternatively, the computer program, when executed by a processor, performs the functions of the modules/units in the above-described apparatus embodiments, e.g., modules 201-206 in fig. 2:
the heartbeat synchronization module 201 is configured to synchronize a first heartbeat table of a database primary library to a database standby library to obtain a second heartbeat table;
the first obtaining module 202 is configured to start a first thread in the database standby library, and obtain a current system time of the database standby library through the first thread Cheng Meige for a preset period of time;
the second obtaining module 203 is configured to obtain, according to the preset time period of the first line Cheng Meige, a target time corresponding to a target field in the second heartbeat table;
The delay monitoring module 204 is configured to obtain a master-slave delay time of the database according to the current system time and the target time.
The computer program, when executed by the processor, further implements the active/standby switching module 205 and the index creating module 206 in the above-described apparatus embodiment, specifically please refer to the second embodiment and the related description thereof.
Example IV
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. In the preferred embodiment of the invention, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33 and a transceiver 34.
It will be appreciated by those skilled in the art that the configuration of the electronic device shown in fig. 3 is not limiting of the embodiments of the present invention, and that either a bus-type configuration or a star-type configuration is possible, and that the electronic device 3 may also include more or less other hardware or software than that shown, or a different arrangement of components.
In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit, a programmable gate array, a digital processor, an embedded device, and the like. The electronic device 3 may further include a client device, where the client device includes, but is not limited to, any electronic product that can interact with a client by way of a keyboard, a mouse, a remote control, a touch pad, or a voice control device, such as a personal computer, a tablet computer, a smart phone, a digital camera, etc.
The electronic device 3 is only an example, and other electronic products that may be present in the present invention or may be present in the future, such as those that may be adapted to the present invention, are also included in the scope of the present invention and are incorporated herein by reference.
In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, performs all or part of the steps in the database master-slave delay monitoring method as described. The Memory 31 includes Read-Only Memory (ROM), programmable Read-Only Memory (PROM), erasable programmable Read-Only Memory (EPROM), one-time programmable Read-Only Memory (One-time Programmable Read-Only Memory, OTPROM), electrically erasable rewritable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disc Memory, magnetic tape Memory, or any other medium that can be used for computer-readable carrying or storing data.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created from the use of blockchain nodes, and the like.
The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm and the like. The Blockchain (Blockchain), which is essentially a decentralised database, is a string of data blocks that are generated by cryptographic means in association, each data block containing a batch of information of network transactions for verifying the validity of the information (anti-counterfeiting) and generating the next block. The blockchain may include a blockchain underlying platform, a platform product services layer, an application services layer, and the like.
In some embodiments, the at least one processor 32 is a Control Unit (Control Unit) of the electronic device 3, connects the various components of the entire electronic device 3 using various interfaces and lines, and performs various functions of the electronic device 3 and processes data by running or executing programs or modules stored in the memory 31, and invoking data stored in the memory 31. For example, the at least one processor 32, when executing the computer program stored in the memory, implements all or part of the steps of the database master/slave delay monitoring method described in embodiments of the present invention; or realize all or part of the functions of the database master-slave delay monitoring device. The at least one processor 32 may be comprised of integrated circuits, such as a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functionality, including one or more central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, and the like.
In some embodiments, the at least one communication bus 33 is arranged to enable connected communication between the memory 31 and the at least one processor 32 or the like.
Although not shown, the electronic device 3 may further comprise a power source (such as a battery) for powering the various components, which may preferably be logically connected to the at least one processor 32 via a power management device, such that functions of managing charging, discharging, and power consumption are performed by the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device 3 may further include various sensors, bluetooth modules, wi-Fi modules, etc., which will not be described herein.
The integrated units implemented in the form of software functional modules described above may be stored in a computer readable storage medium. The software functional modules described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, an electronic device, or a network device, etc.) or a processor (processor) to perform portions of the methods described in the various embodiments of the invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division when actually implemented.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it will be obvious that the term "comprising" does not exclude other elements or that the singular does not exclude a plurality. Several of the elements or devices recited in the specification may be embodied by one and the same item of software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.

Claims (10)

1. A method for monitoring a database master-slave delay, the method comprising:
synchronizing the first heartbeat table of the database main library into the database standby library to obtain a second heartbeat table;
starting a first thread in the database standby library, and acquiring the current system time of the database standby library through a preset time period of the first thread Cheng Meige;
acquiring a target time corresponding to a target field in the second heartbeat table through the preset time period of the first line Cheng Meige, wherein the target time is the current system time of the database main library, and the time format of the target field is accurate to microseconds;
and obtaining the main and standby delay time of the database according to the current system time and the target time.
2. The database primary and backup delay monitoring method of claim 1, wherein prior to said synchronizing the first heartbeat table of the database primary library into the database backup library, the method further comprises:
Creating the first heartbeat table in the database main library;
starting a second thread in the database main library, and writing the current system time of the database main library into the target field in the first heartbeat table through the second thread;
updating the current system time of the database main library every other preset time period through the second thread;
and synchronizing the updated current system time in the database main library to the target field of the database standby library.
3. The method of claim 2, wherein updating the current system time of the database master every the preset time period by the second thread comprises:
and updating the current system time of the database main library in the memory at intervals of the preset time period through the second thread.
4. The database master/slave delay monitoring method of claim 1, further comprising:
acquiring process state information and health state information of the database main library;
judging whether the database main library has faults or not according to the process state information and the health state information;
And if the database main library is determined to have faults, controlling the database main library and the database standby library to perform main-standby switching operation.
5. The method for monitoring the delay of primary and secondary databases of claim 4, wherein said determining whether the primary database fails according to the process state information and the health state information comprises:
if the process of the database main library is determined to be absent, and the read health state information is information representing that the database main library is in an unhealthy state currently, determining that the database main library fails;
and if the process of the database main library is determined to exist, or the read health state information is information representing that the database main library is in a health state currently, determining that the database main library has no fault.
6. A method of monitoring a database master-slave delay according to any one of claims 1 to 5, the method further comprising:
generating a database log;
storing user operations on the database master library in the database log;
calculating the total times of the user operation in a preset time period and the times of target operation on each column in the database main library;
Generating an index column name according to the total times of the user operation and the times of target operation on each column in the database main library;
creating an index of the database main library according to the index column name in a preset period;
the preset time period is a time period with the lowest database access amount in the preset time period.
7. The method of claim 6, wherein generating an index column name based on the total number of user operations and the number of target operations performed on each column in the database master library comprises:
calculating to obtain the number of times of target operation according to the total number of times of user operation and the number of times of target operation on each column in the database main library;
judging whether the frequency duty ratio is higher than a preset duty ratio threshold value or not;
when the frequency occupation ratio is higher than the preset occupation ratio threshold value, creating an index column name for a column corresponding to the target operation in the database main library;
the remaining columns in the database master library do not create index column names.
8. A database master-slave delay monitoring apparatus, the apparatus comprising:
the heartbeat synchronization module is used for synchronizing the first heartbeat table of the database main library into the database standby library to obtain a second heartbeat table;
The first obtaining module is used for starting a first thread in the database standby library and obtaining the current system time of the database standby library through the first thread Cheng Meige in a preset time period;
the second obtaining module is configured to obtain, according to the preset time period of the first line Cheng Meige, a target time corresponding to a target field in the second heartbeat table, where the target time is a current system time of the database main library, and a time format of the target field is accurate to microseconds;
and the delay monitoring module is used for obtaining the main and standby delay time of the database according to the current system time and the target time.
9. An electronic device, characterized in that the electronic device comprises a processor and a memory, the processor being adapted to implement the database master/slave delay monitoring method according to any one of claims 1 to 7 when executing a computer program stored in the memory.
10. A computer readable storage medium having a computer program stored thereon, wherein the computer program when executed by a processor implements the database master backup delay monitoring method according to any of claims 1 to 7.
CN202311037719.8A 2023-08-16 2023-08-16 Database master-slave delay monitoring method and device, electronic equipment and storage medium Pending CN116827828A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311037719.8A CN116827828A (en) 2023-08-16 2023-08-16 Database master-slave delay monitoring method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311037719.8A CN116827828A (en) 2023-08-16 2023-08-16 Database master-slave delay monitoring method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116827828A true CN116827828A (en) 2023-09-29

Family

ID=88120537

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311037719.8A Pending CN116827828A (en) 2023-08-16 2023-08-16 Database master-slave delay monitoring method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116827828A (en)

Similar Documents

Publication Publication Date Title
US9141685B2 (en) Front end and backend replicated storage
Nawab et al. Minimizing commit latency of transactions in geo-replicated data stores
US8301600B1 (en) Failover recovery in a distributed data store
US10726042B2 (en) Replication control using eventually consistent meta-data
US20180004777A1 (en) Data distribution across nodes of a distributed database base system
Yan et al. Carousel: Low-latency transaction processing for globally-distributed data
JP5308403B2 (en) Data processing failure recovery method, system and program
US20130246358A1 (en) Online verification of a standby database in log shipping physical replication environments
US9262279B2 (en) Classifying and monitoring database operations based on a cost of recovery
CN110019469A (en) Distributed data base data processing method, device, storage medium and electronic device
WO2018234265A1 (en) System and apparatus for a guaranteed exactly once processing of an event in a distributed event-driven environment
US20240004746A1 (en) Access Consistency in High-Availability Databases
CN112527568A (en) Data flow testing method and device, electronic equipment and storage medium
CN116827828A (en) Database master-slave delay monitoring method and device, electronic equipment and storage medium
CN113537507A (en) Machine learning system, method and electronic equipment
CN107590286B (en) Method and device for managing transaction information in cluster file system
CN111400098A (en) Copy management method and device, electronic equipment and storage medium
CN117851520B (en) Data synchronization method, system, equipment and medium of securities core transaction engine
US11663098B2 (en) Maintaining durability of a data object using unplanned delta components during transient failures
RU2714602C1 (en) Method and system for data processing
CN115455331A (en) Page table single data processing method and device, electronic equipment and storage medium
CN115237890A (en) Data migration method, device, medium and equipment for database splitting
WO2023198276A1 (en) Handling failure of an application instance
Zakhary et al. A System Infrastructure for Strongly Consistent Transactions on Globally-Replicated Data
Agrawal et al. Minimizing Commit Latency of Transactions in Geo-Replicated Data Stores

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination