CN113760872A - Database monitoring method and device and computer readable storage medium - Google Patents
Database monitoring method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN113760872A CN113760872A CN202010909421.1A CN202010909421A CN113760872A CN 113760872 A CN113760872 A CN 113760872A CN 202010909421 A CN202010909421 A CN 202010909421A CN 113760872 A CN113760872 A CN 113760872A
- Authority
- CN
- China
- Prior art keywords
- database
- snapshot information
- monitoring
- time
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 126
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000008859 change Effects 0.000 claims abstract description 31
- 238000012217 deletion Methods 0.000 claims description 20
- 230000037430 deletion Effects 0.000 claims description 20
- 238000012986 modification Methods 0.000 claims description 13
- 230000004048 modification Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The disclosure relates to a database monitoring method and device and a computer readable storage medium. The database monitoring method comprises the following steps: responding to a database monitoring request, and starting a database monitoring task according to monitoring configuration information included in the database monitoring request; collecting database snapshot information in a specified time period according to a database monitoring task; storing the collected database snapshot information; judging whether at least one of table data change and database master-slave delay occurs to the monitored database according to the stored database snapshot information; and determining whether to start alarming according to the judgment result.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a database monitoring method and apparatus, and a computer-readable storage medium.
Background
Software is typically comprised of programs, data, and documents. A document is a description of software by a developer or a user. A program is business logic code that manages data. Data occupies a significant position throughout the software.
Most of the data in the software needs to be stored permanently so that the application can still read the previous data after restart or failure recovery. Permanently storing data on a machine is storing data in the form of files on a computer's disk. Based on this, a large number of relational databases are produced. Software queries the data on the database through Structured Query Language (SQL).
Monitoring of the database is mainly limited to monitoring of a physical machine of the database, for example, information such as a Central Processing Unit (CPU), a disk, an input/output Interface (IO), a database connection number and the like of the database is monitored, and data in the database of the biased service is rarely monitored. Therefore, when the software is subjected to risks such as data loss or physical deletion, the user cannot find problems in time, and timely recovery cannot be guaranteed. In the related art of database monitoring, data monitoring is realized by developing a custom monitoring tool.
Disclosure of Invention
According to some embodiments of the present disclosure, there is provided a database monitoring method including:
responding to a database monitoring request, and starting a database monitoring task according to monitoring configuration information included in the database monitoring request;
collecting database snapshot information collected in a specified time period according to a database monitoring task;
storing the collected database snapshot information;
judging whether at least one of table data change and database master-slave delay occurs to the monitored database according to the stored database snapshot information;
and determining whether to start alarming according to the judgment result.
In some embodiments, the database snapshot information includes: at least one of the total number of data records, the number of valid data records, the number of logical deletion records, the maximum record primary key ID, the maximum creation time and the maximum modification time of a certain data table at the acquisition time.
In some embodiments, in the case that database snapshot information of a certain data table changes within a specified time period, it is determined that a table data change has occurred to the data table.
In some embodiments, the table data change includes a physical deletion of a database table, and for a data table, the data table is determined to have the physical deletion of the database table if the database snapshot information satisfies any one of the following conditions:
the total number of data records in the database snapshot information acquired this time is less than the total number of data records in the database snapshot information acquired last time;
the maximum record main key ID in the database snapshot information collected this time is smaller than the maximum record main key ID in the database snapshot information collected last time;
the number of the logic deletion records in the snapshot information of the database collected this time is smaller than the number of the logic deletion records in the snapshot information of the monitoring result database at the last time;
the maximum creation time in the database snapshot information collected this time is less than the maximum creation time in the database snapshot information collected last time;
the maximum modification time in the database snapshot information collected this time is less than the maximum modification time in the database snapshot information collected last time;
the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum creation time in the database snapshot information acquired this time is longer than the maximum creation time in the database snapshot information acquired last time;
the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum record primary key ID in the database snapshot information acquired this time is greater than the maximum record primary key ID in the database snapshot information acquired last time.
In some embodiments, for the same acquisition time, in the case that the database snapshot information of the database master library and the database slave library satisfies any one of the following conditions, it is determined that the database master-slave delay occurs:
the difference value of the maximum record primary key ID in the database snapshot information of the database master library and the database slave library is greater than a first threshold value;
the difference value of the maximum creating time in the database snapshot information of the database master library and the database slave library is larger than a second threshold value;
the difference of the maximum modification time in the database snapshot information of the database master library and the slave library is larger than a third threshold value.
In some embodiments, determining, for the monitored database, based on the stored information, whether at least one of a table data change, a database master slave delay, occurs comprises:
presenting data in the stored database snapshot information in a chart mode;
and judging whether at least one of table data change and database master-slave delay occurs according to fluctuation of the chart.
In some embodiments, determining whether to activate an alarm based on the result of the determination includes:
counting data change information in a specified time period according to the stored database snapshot information, wherein the data change information comprises the variable quantity of a database table;
and determining to start the alarm under the condition that the statistical variation is larger than or equal to the alarm threshold value in the monitoring configuration information.
In some embodiments, the data change information includes: and the logic of one data table deletes at least one item of the number of the items and modifies at least one item of the number of the items in the specified time period.
In some embodiments, collecting database snapshot information over a specified time period in accordance with the database monitoring task comprises: and querying the database snapshot information in the specified time period through a Structured Query Language (SQL).
In some embodiments, database snapshot information is stored using Redis distributed caching.
In some embodiments, monitoring configuration information comprises: at least one of the name of the database to be monitored, the name of the data table to be monitored, the monitoring frequency, the configuration of the database data source and the configuration of the monitoring alarm.
According to further embodiments of the present disclosure, there is provided a database monitoring apparatus including:
the starting unit is configured to respond to the database monitoring request and start a database monitoring task according to monitoring configuration information included in the database monitoring request;
the acquisition unit is configured to acquire database snapshot information acquired within a specified time period according to the database monitoring task;
the storage unit is configured to store the acquired database snapshot information;
the judging unit is configured to judge whether at least one of table data change and database master-slave delay occurs to the monitored database according to the stored database snapshot information;
and the determining unit is configured to determine whether to start an alarm according to the judgment result.
According to still further embodiments of the present disclosure, there is provided a database monitoring apparatus including: a memory; and a processor coupled to the memory, the processor configured to perform the database monitoring method of any of the above embodiments based on instructions stored in the memory device.
According to still further embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the database monitoring method in any of the above embodiments.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
The present disclosure may be understood more clearly and in accordance with the following detailed description, taken with reference to the accompanying drawings,
wherein:
FIG. 1 illustrates a flow diagram of a database monitoring method according to some embodiments of the present disclosure;
FIG. 2 illustrates a flow diagram of a database monitoring method according to further embodiments of the present disclosure;
FIG. 3 illustrates a flow diagram of a database monitoring method according to further embodiments of the present disclosure;
FIG. 4 illustrates a block diagram of a database monitoring apparatus, according to some embodiments of the present disclosure;
FIG. 5 is a block diagram illustrating a database monitoring apparatus according to further embodiments of the present disclosure;
FIG. 6 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
The inventor finds out through research that: for custom database monitoring, once the monitoring requirements change, the originally customized code will be difficult to adapt to the new monitoring requirements. In addition, most of the existing database monitoring functions are developed according to the binlog of the database, and the functions are seriously dependent on whether the database supports the binlog, whether the binlog is opened, the integrity of the binlog file, and even the matching degree of the format of the binlog and the version of the database. In addition, parsing the binlog file itself is a relatively complex matter. The database monitoring mode is not enough to meet the requirements of practical application in the aspects of universality, accuracy and the like.
In order to solve the above problems, the present disclosure provides a database monitoring scheme, which can meet the monitoring requirements of different databases.
FIG. 1 is a flow diagram illustrating a database monitoring method according to some embodiments of the present disclosure. As shown in FIG. 1, the database monitoring method includes steps S1-S5.
In step S1, in response to the database monitoring request, a database monitoring task is started according to the monitoring configuration information included in the database monitoring request.
In some embodiments, monitoring configuration information comprises: at least one of the name of the database to be monitored, the name of the data table to be monitored, the monitoring frequency, the configuration of the database data source and the configuration of the monitoring alarm. The monitoring alarm configuration comprises data change alarm configuration, database master-slave delay alarm configuration and the like.
The monitoring configuration information may be configured by a monitoring configuration module. The monitoring configuration information may also include database table alias configuration.
The database monitoring task may be initiated by a monitoring scheduling module. In some embodiments, the monitoring scheduling module starts the timing detection task according to the monitoring frequency in the monitoring configuration information. And if the monitoring frequency in the monitoring configuration information changes, stopping the previous scheduling by the monitoring scheduling module and performing the scheduling again at the new monitoring frequency.
In step S2, database snapshot information is collected over a specified time period in accordance with the database monitoring task.
In some embodiments, the data snapshot information includes: at the time of collection, at least one of the total number of data records, the number of valid data records (such as the number of data records in the data table which are not logically deleted), the number of logically deleted records, the maximum record primary key ID, the maximum creation time and the maximum modification time of a certain data table.
The database snapshot information may be collected by a data collection module. In some embodiments, the database snapshot information over a specified time period is queried via Structured Query Language (SQL). For example, SQL statements are executed on a monitored database through the JDBC API.
In step S3, the collected database snapshot information is stored.
In some embodiments, the collected database snapshot information is stored to memory and a distributed cache. The data structure stored in the memory is, for example, a Map set, in which Key represents a data table name and Value represents data snapshot information, and the read library structure and the write library structure are stored separately. Therefore, the database snapshot information collected last time can be recorded in the memory.
In some embodiments, database snapshot information is stored using Redis distributed caching. The data structure used for storing the data snapshot information in the Redis distributed cache is SortedSet, and the data structure is an ordered Set. This data structure stores, in addition to data, time information representing the data storage this time. For example, for the current time, hours, minutes, seconds are taken and a description (hours 3600+ minutes 60+ seconds) representing the current time is calculated. The obtained storage data can be stored according to the time sequence, and the acquisition information in the appointed time period can be very simply acquired in the follow-up process.
In step S4, it is determined whether at least one of a change in table data and a database master-slave delay has occurred with respect to the monitored database based on the stored database snapshot information.
In some embodiments, in the case that database snapshot information of a certain data table changes within a specified time period, it is determined that a table data change has occurred to the data table.
The table data change includes a database table physical deletion. According to the stored database snapshot information, whether the database table physical deletion occurs can be judged. In some embodiments, for a data table, a determination is made that a physical deletion of the data table has occurred in the data table if the database snapshot information satisfies any of the following conditions:
1) the total number of data records in the database snapshot information acquired this time is less than the total number of data records in the database snapshot information acquired last time;
2) the maximum record main key ID in the database snapshot information collected this time is smaller than the maximum record main key ID in the database snapshot information collected last time;
3) the number of the logic deletion records in the snapshot information of the database collected this time is smaller than the number of the logic deletion records in the snapshot information of the monitoring result database at the last time;
4) the maximum creation time in the database snapshot information collected this time is less than the maximum creation time in the database snapshot information collected last time;
5) the maximum modification time in the database snapshot information collected this time is less than the maximum modification time in the database snapshot information collected last time;
6) the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum creation time in the database snapshot information acquired this time is longer than the maximum creation time in the database snapshot information acquired last time;
7) the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum record primary key ID in the database snapshot information acquired this time is greater than the maximum record primary key ID in the database snapshot information acquired last time.
And according to the stored database snapshot information, whether the master-slave delay of the database occurs can be judged. In some embodiments, for the same acquisition time, in the case that the database snapshot information of the database master library and the database slave library satisfies any one of the following conditions, it is determined that the database master-slave delay occurs:
1) the difference value of the maximum record primary key ID in the database snapshot information of the database master library and the database slave library is greater than a first threshold value;
2) the difference value of the maximum creating time in the database snapshot information of the database master library and the database slave library is larger than a second threshold value;
3) the difference of the maximum modification time in the database snapshot information of the database master library and the slave library is larger than a third threshold value.
The first threshold, the second threshold, and the third threshold may be included in the monitoring configuration information. And the first threshold, the second threshold and the third threshold can be flexibly configured according to different service requirements.
In some embodiments, it may also be visually determined whether a database table physical deletion or a database master-slave delay has occurred using the database snapshot information. How to determine whether a database table physical deletion or a database master slave delay occurs is described below in conjunction with fig. 2.
As shown in fig. 2, the flow of determination includes: step S41, presenting the data in the stored database snapshot information in a chart mode; and step S42, judging whether the database table physical deletion or the database master-slave delay occurs according to the fluctuation of the chart.
For example, the monitoring result data is queried from the distributed cache by the monitoring data reporting module, and since the monitoring is continuous, the data in the database snapshot information can be presented in the form of a line graph (for example, a connection line of data points at different times). If a downward inflection point appears in the line graph, it indicates that physical deletion of the database has occurred. If large fluctuation occurs in the database report, the information is also prompted to pay attention to whether the abnormality occurs. If the points in the line graphs of the master-slave database do not coincide with each other or the difference value of the points in the same time exceeds a threshold value, the master-slave delay of the database can be considered to occur. In this way, problems can be found intuitively through database reports.
As described above, the stored information may include data change information in addition to the database snapshot information, and may be used to determine whether a table data change has occurred.
The storing of the information in step S3 and the determining in step S4 may be performed by the monitoring analysis module. The monitoring analysis module is used for monitoring data change of the table and master-slave delay of the database and storing a monitoring result of each analysis.
Next, in step S5, it is determined whether or not to activate an alarm, based on the result of the determination. How to determine whether to activate an alarm is described below in conjunction with fig. 3.
As shown in fig. 3, the process of determining whether to activate an alarm includes: step S51, according to the stored database snapshot information, counting the data change information in the appointed time period, wherein the data change information includes the variation of the database table; and step S52, determining to start alarm when the statistical variation is larger than or equal to the alarm threshold value in the monitoring configuration information.
In some embodiments, the data change information includes: and the logic of one data table deletes at least one item of the number of the items and modifies at least one item of the number of the items in the specified time period.
Once the alarm is determined to be started, the data alarm module sends alarm information to the user, for example, the alarm information is sent by means of mails, short messages, instant messaging messages and the like. The alarm information may include, for example: IP address of the database, database name, data table name, table alias information, alarm detail information, and the like.
In the embodiment, by using the dynamic configuration, the hard coding and the defect that the interrupted application program needs to be restarted when the configuration is modified conventionally can be solved. According to the dynamic configuration of the alarm threshold value, dynamic alarm can be realized, and the data condition of the database can be known in time.
According to some embodiments of the present disclosure, a database monitoring apparatus is further provided, which can implement the database monitoring method described in any of the above embodiments.
FIG. 4 illustrates a block diagram of a database monitoring apparatus, according to some embodiments of the present disclosure.
As shown in fig. 4, the database monitoring apparatus 40 includes a starting unit 410, an acquisition unit 420, a storage unit 430, a judgment unit 440, and a determination unit 450.
The initiating unit 410 is configured to initiate a database monitoring task in response to the database monitoring request according to the monitoring configuration information included in the database monitoring request, for example, to perform step S1.
The collecting unit 420 is configured to collect database snapshot information within a specified time period according to the database monitoring task, for example, perform step S2.
The storage unit 430 is configured to store the collected database snapshot information, for example, to perform step S3.
The determining unit 440 is configured to determine whether at least one of table data change and database master-slave delay occurs for the monitored database according to the stored database snapshot information, for example, execute step S4.
The determination unit 450 is configured to determine whether to activate an alarm according to the result of the determination, for example, to perform step S5.
FIG. 5 illustrates a block diagram of a database monitoring apparatus, according to further embodiments of the present disclosure.
As shown in fig. 5, the database monitoring apparatus 50 includes: a memory 510 and a processor 520 coupled to the memory 510, the processor 520 configured to perform one or more steps of a database monitoring method in any of the embodiments of the present disclosure based on instructions stored in the memory 510.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, apparatus, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Accordingly, embodiments of the present disclosure also provide a computer readable storage medium having stored thereon computer instructions, which when executed by a processor, implement one or more steps of the database monitoring method in any of the foregoing embodiments.
FIG. 6 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
As shown in FIG. 6, the computer system may be embodied in the form of a general purpose computing device, which may be used to implement the handwritten text recognition apparatus of the above-described embodiments. The computer system includes a memory 610, a processor 620, and a bus 600 that connects the various system components.
The memory 610 may include, for example, system memory, non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), and other programs. The system memory may include volatile storage media such as Random Access Memory (RAM) and/or cache memory. The non-volatile storage medium stores, for example, instructions to perform corresponding embodiments of the display method. Non-volatile storage media include, but are not limited to, magnetic disk storage, optical storage, flash memory, and the like.
The processor 620 may be implemented as discrete hardware components, such as a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gates or transistors, or the like. Accordingly, each device, such as the determination device and the determination device, may be realized by a Central Processing Unit (CPU) executing instructions in a memory for performing the corresponding steps, or may be realized by a dedicated circuit for performing the corresponding steps.
The computer system may also include an input-output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650 and the memory 610 and the processor 620 may be connected by a bus 600. The input/output interface 630 may provide a connection interface for input/output devices such as a display, a mouse, and a keyboard. The network interface 640 provides a connection interface for various networking devices. The storage interface 640 provides a connection interface for external storage devices such as a floppy disk, a usb disk, and an SD card.
Thus, database monitoring methods, apparatus, and systems, and computer-readable storage media according to the present disclosure have been described in detail. Some details that are well known in the art have not been described in order to avoid obscuring the concepts of the present disclosure. It will be fully apparent to those skilled in the art from the foregoing description how to practice the presently disclosed embodiments.
The method and system of the present disclosure may be implemented in a number of ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the foregoing examples are for purposes of illustration only and are not intended to limit the scope of the present disclosure. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims (14)
1. A database monitoring method, comprising:
responding to a database monitoring request, and starting a database monitoring task according to monitoring configuration information included in the database monitoring request;
collecting database snapshot information in a specified time period according to a database monitoring task;
storing the collected database snapshot information;
judging whether at least one of table data change and database master-slave delay occurs to the monitored database according to the stored database snapshot information;
and determining whether to start alarming according to the judgment result.
2. The database monitoring method according to claim 1, wherein the database snapshot information comprises: at least one of the total number of data records, the number of valid data records, the number of logical deletion records, the maximum record primary key ID, the maximum creation time and the maximum modification time of a certain data table at the acquisition time.
3. The database monitoring method according to claim 1, wherein in a case where database snapshot information of a certain data table is changed within a specified time period, it is judged that a table data change has occurred to the data table.
4. The database monitoring method according to claim 3, wherein the table data change includes a physical deletion of the database table, and for a database table, the physical deletion of the database table is determined to have occurred in the database table if the database snapshot information satisfies any one of the following conditions:
the total number of data records in the database snapshot information acquired this time is less than the total number of data records in the database snapshot information acquired last time;
the maximum record main key ID in the database snapshot information collected this time is smaller than the maximum record main key ID in the database snapshot information collected last time;
the number of the logic deletion records in the snapshot information of the database collected this time is smaller than the number of the logic deletion records in the snapshot information of the monitoring result database at the last time;
the maximum creation time in the database snapshot information collected this time is less than the maximum creation time in the database snapshot information collected last time;
the maximum modification time in the database snapshot information collected this time is less than the maximum modification time in the database snapshot information collected last time;
the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum creation time in the database snapshot information acquired this time is longer than the maximum creation time in the database snapshot information acquired last time;
the total number of data records in the database snapshot information acquired this time is the same as the total number of data records in the database snapshot information acquired last time, and the maximum record primary key ID in the database snapshot information acquired this time is greater than the maximum record primary key ID in the database snapshot information acquired last time.
5. The database monitoring method according to claim 2, wherein, for the same acquisition time, in case that the database snapshot information of the database master library and the database slave library satisfies any one of the following conditions, it is determined that the database master-slave delay has occurred:
the difference value of the maximum record primary key ID in the database snapshot information of the database master library and the database slave library is greater than a first threshold value;
the difference value of the maximum creating time in the database snapshot information of the database master library and the database slave library is larger than a second threshold value;
the difference of the maximum modification time in the database snapshot information of the database master library and the slave library is larger than a third threshold value.
6. The database monitoring method according to claim 2, wherein determining, for the monitored database, whether at least one of a change in table data, a database master slave delay, etc., has occurred based on the stored information comprises:
presenting data in the stored database snapshot information in a chart mode;
and judging whether at least one of table data change and database master-slave delay occurs according to fluctuation of the chart.
7. The database monitoring method according to any one of claims 1 to 6, wherein determining whether to activate an alarm according to a result of the determination includes:
counting data change information in a specified time period according to the stored database snapshot information, wherein the data change information comprises the variable quantity of a database table;
and determining to start the alarm under the condition that the statistical variation is larger than or equal to the alarm threshold value in the monitoring configuration information.
8. The database monitoring method according to claim 7, wherein the data change information further comprises: and the logic of one data table deletes at least one item of the number of the items and modifies at least one item of the number of the items in the specified time period.
9. The database monitoring method according to any one of claims 1 to 6, wherein collecting database snapshot information within a specified time period according to a database monitoring task comprises:
and querying the database snapshot information in the specified time period through a Structured Query Language (SQL).
10. The database monitoring method according to any of claims 1 to 6, wherein the database snapshot information is stored using Redis distributed caching.
11. The database monitoring method according to any one of claims 1 to 6, wherein monitoring configuration information comprises: at least one of the name of the database to be monitored, the name of the data table to be monitored, the monitoring frequency, the configuration of the database data source and the configuration of the monitoring alarm.
12. A database monitoring apparatus comprising:
the starting unit is configured to respond to the database monitoring request and start a database monitoring task according to monitoring configuration information included in the database monitoring request;
the acquisition unit is configured to acquire database snapshot information acquired within a specified time period according to the database monitoring task;
the storage unit is configured to store the acquired database snapshot information;
the judging unit is configured to judge whether at least one of table data change and database master-slave delay occurs to the monitored database according to the stored database snapshot information;
and the determining unit is configured to determine whether to start an alarm according to the judgment result.
13. A database monitoring apparatus comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the database monitoring method of any of claims 1 to 11 based on instructions stored in the memory device.
14. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the database monitoring method according to any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010909421.1A CN113760872A (en) | 2020-09-02 | 2020-09-02 | Database monitoring method and device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010909421.1A CN113760872A (en) | 2020-09-02 | 2020-09-02 | Database monitoring method and device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113760872A true CN113760872A (en) | 2021-12-07 |
Family
ID=78785776
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010909421.1A Pending CN113760872A (en) | 2020-09-02 | 2020-09-02 | Database monitoring method and device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760872A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115510144A (en) * | 2022-11-17 | 2022-12-23 | 北京滴普科技有限公司 | Method and system for capturing real-time change data of database |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130091181A1 (en) * | 2011-10-05 | 2013-04-11 | International Business Machines Corporation | Monitoring stored procedure execution |
| CN106156047A (en) * | 2015-03-27 | 2016-11-23 | 中国移动通信集团福建有限公司 | A kind of SNAPSHOT INFO processing method and processing device |
| CN110334150A (en) * | 2019-06-05 | 2019-10-15 | 上海易点时空网络有限公司 | Monitor method, monitor terminal and the system of master-slave database synchronizing relay |
-
2020
- 2020-09-02 CN CN202010909421.1A patent/CN113760872A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130091181A1 (en) * | 2011-10-05 | 2013-04-11 | International Business Machines Corporation | Monitoring stored procedure execution |
| CN106156047A (en) * | 2015-03-27 | 2016-11-23 | 中国移动通信集团福建有限公司 | A kind of SNAPSHOT INFO processing method and processing device |
| CN110334150A (en) * | 2019-06-05 | 2019-10-15 | 上海易点时空网络有限公司 | Monitor method, monitor terminal and the system of master-slave database synchronizing relay |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115510144A (en) * | 2022-11-17 | 2022-12-23 | 北京滴普科技有限公司 | Method and system for capturing real-time change data of database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108959564B (en) | Data warehouse metadata management method, readable storage medium and computer device | |
| JP6048038B2 (en) | Information processing apparatus, program, and information processing method | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| US20190095391A1 (en) | Processing a data set | |
| CN112100138A (en) | Log query method and device, storage medium and electronic equipment | |
| US9678970B2 (en) | Database storage reclaiming program | |
| CN113760872A (en) | Database monitoring method and device and computer readable storage medium | |
| US20160098473A1 (en) | Grouping method and apparatus | |
| CN111413952B (en) | Robot fault detection method and device, electronic equipment and readable storage medium | |
| CN111831528A (en) | Computer system log association method and related device | |
| US11023449B2 (en) | Method and system to search logs that contain a massive number of entries | |
| US11010158B2 (en) | Determining the availability of memory optimizations by analyzing a running binary | |
| CN107402920B (en) | Method and device for determining correlation complexity of relational database table | |
| CN114077532A (en) | SQL statement execution efficiency detection method and device | |
| CN114090673A (en) | Data processing method, equipment and storage medium for multiple data sources | |
| CN114461762A (en) | Archive change identification method, device, equipment and storage medium | |
| CN112948478A (en) | Link-based code analysis method and device, electronic equipment and storage medium | |
| CN112241262A (en) | Software-defined satellite-oriented reusable code extracting, analyzing and retrieving method and device | |
| CN112783711A (en) | Method and storage medium for analyzing program memory on NodeJS | |
| CN113127427B (en) | Method, system and device for analyzing transaction distribution in database log | |
| CN113434509B (en) | Increment index updating method and device, storage medium and electronic equipment | |
| CN111625853B (en) | Snapshot processing method, device and equipment and readable storage medium | |
| CN117762969A (en) | Data sampling method, device, equipment, medium and product | |
| CN112612682A (en) | Method, system and device for dynamically monitoring file and storage medium | |
| CN117909392A (en) | Intelligent data asset inventory method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |