CN113553382A - Data page disk refreshing method, device, equipment and storage medium - Google Patents
Data page disk refreshing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113553382A CN113553382A CN202110901940.8A CN202110901940A CN113553382A CN 113553382 A CN113553382 A CN 113553382A CN 202110901940 A CN202110901940 A CN 202110901940A CN 113553382 A CN113553382 A CN 113553382A
- Authority
- CN
- China
- Prior art keywords
- log
- sequence
- log sequence
- packet
- page
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a method, a device and equipment for refreshing a data page and a storage medium. The method is applied to a database cluster, wherein the database cluster comprises a main library and at least two standby libraries; the method comprises the following steps: the master library sends heartbeat messages or generated log packets to the standby libraries; the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed; the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence; and if the disk refreshing condition is met, writing the data in the data page to be refreshed into a local disk. The problems of insufficient data page caching and replay blocking of the standby database can be avoided, and the database cluster can be ensured to normally and stably operate.
Description
Technical Field
The embodiment of the invention relates to the technical field of databases, in particular to a method, a device, equipment and a storage medium for refreshing a data page.
Background
With the rapid development of information technology, databases play an extremely important role in daily work, business activities, and people's lives. In order to avoid the problem that the databases cannot normally provide services due to downtime in the using process, a database cluster building mode is often adopted in the prior application, namely, a plurality of databases are divided into a main database and a standby database, so that the standby database can still continue to provide services when the main database crashes.
In order to ensure data synchronization of each database in the database cluster, a log packet is generated when data modification operations such as addition, deletion and modification are carried out on the main database, and the log packet is sent to the standby database, so that the standby database can replay according to the log packet and further keep data synchronization with the main database.
In the prior art, when a packet sequence number or a log sequence of a log packet being replayed by a standby library is smaller than a packet sequence number or a log sequence already submitted by a main library, at this time, the standby library needs to allocate a new data page for replay, but the standby library has a situation that a data page cache is insufficient and cannot be allocated, the standby library tries to flush the allocated data page so as to eliminate an available data page cache, but because the submitted log packet sequence number and the submitted log sequence that the standby library advances following the log packet replay, the standby library is limited by the submitted log sequence when trying to flush the data page, so that a data page cannot be flushed, and the situation that the standby library cannot allocate a new data page and cannot continue to replay, so that the standby library is abnormal and a read-only service cannot be provided to the outside occurs.
Disclosure of Invention
The embodiment of the invention provides a method, a device and equipment for refreshing a data page and a storage medium. The problems of insufficient data page caching and replay blocking of the standby database can be avoided, and the database cluster can be ensured to normally and stably operate.
In a first aspect, an embodiment of the present invention provides a data page disk-flushing method, which is applied to a database cluster, where the database cluster includes a master library and at least two backup libraries; the method comprises the following steps:
the master library sends heartbeat messages or generated log packets to the standby libraries; wherein the heartbeat message or the log packet carries a first submitted log sequence;
the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed;
the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence;
and if the disk refreshing condition is met, writing the data in the data page to be refreshed into a local disk.
In a second aspect, an embodiment of the present invention further provides a data page disk-flushing device, which is disposed in a database cluster, where the database cluster includes a master library and at least two backup libraries; the device comprises:
the heartbeat message or log packet sending module is arranged in the main library and is used for sending the heartbeat message or the generated log packet to each standby library; wherein the heartbeat message or the log packet carries a first submitted log sequence;
the log sequence acquisition module is arranged in the standby library and used for extracting a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk and acquiring a page log sequence in a current data page to be refreshed;
the disk-brushing condition judging module is arranged in the standby library and used for judging whether the data pages to be brushed meet the disk-brushing conditions or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence;
and the data page disk-brushing module is used for writing the data in the data page to be brushed into the local disk when the disk-brushing condition is met.
In a third aspect, an embodiment of the present invention further provides an electronic device, including: a database cluster, storage, and one or more processors;
the database cluster comprises a main library and at least two standby libraries;
the storage device to store one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for flushing the data page according to the embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method for flushing a data page according to the embodiment of the present invention.
The embodiment of the invention discloses a method, a device and equipment for refreshing a data page and a storage medium. The method is applied to a database cluster, wherein the database cluster comprises a main library and at least two standby libraries; the method comprises the following steps: the master library sends the heartbeat message or the generated log packet to each standby library; the heartbeat message or the log packet carries a first submitted log sequence; the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed; the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence; and if the disk refreshing condition is met, writing the data in the data page to be refreshed into the local disk. According to the data page disk-refreshing method provided by the embodiment of the invention, whether the data page to be refreshed meets the disk-refreshing condition is judged according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence, more memory space can be released in time to allocate new data page cache, the problems of insufficient data page cache and replay blockage of a standby database can be avoided, and the database cluster can be ensured to normally and stably run.
Drawings
FIG. 1 is a flowchart of a method for flushing a data page in a first embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a disk brushing device for data pages according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device in a third embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
In the related technology, the main library and the standby library respectively advance a submitted log sequence (C _ LSN) and a submitted log packet sequence number (C _ SEQNO) through synchronous log packets, data PAGE flushing actions of the main library and the standby library are limited through the submitted log sequence, and data PAGE flushing is allowed only when the PAGE log sequence (PAGE _ LSN) of data PAGEs is less than or equal to the submitted log sequence, so that logs larger than the submitted log sequence can be safely truncated.
As long as the log packets generated by the master library are written into more than a set number of databases (including the master library), C _ SEQNO and C _ LSN can be pushed forward, a small number of spare libraries with slower replay do not influence the normal operation of the master library, and the C _ SEQNO and C _ LSN of the spare libraries are pushed forward depending on the C _ SEQNO and C _ LSN filled in the log packets of the master library. However, for a few spare libraries, C _ SEQNO and C _ LSN on the currently replayed log packet may be smaller than C _ SEQNO and C _ LSN already pushed by the main library, and if insufficient data PAGE cache (new data PAGE cache cannot be allocated) occurs on these spare libraries, the spare libraries may try to eliminate the allocated data PAGEs from the disk, but since the C _ LSN that the spare libraries push along with the log packet replay is smaller, the C _ LSN is limited by the value of C _ LSN when trying to eliminate the data PAGEs, the data PAGEs whose PAGE _ LSN is greater than C _ LSN cannot be flushed, so that the spare libraries cannot continue to replay, and the condition that the spare libraries system is abnormal and cannot provide read-only service to the outside occurs.
As an example, it is assumed that a certain database cluster includes 3 databases, EP1, EP2, and EP3, where EP1 is the master database, EP2 and EP3 are the standby databases, the current C _ seq no ═ 2999, and C _ LSN ═ 5000 of the master database.
1. EP1 generates a new log packet (RPKG1), which is RPKG1 the information filled in the log packet: c _ SEQNO ═ 2999, C _ LSN ═ 5000, G _ SEQNO ═ 3000 (new log packet sequence number), and MAX _ LSN ═ 8000 (maximum log sequence in new log packets).
EP1 sends log packets (RPKG1) to EP2 and EP 3.
After receiving the log packet (RPKG1), EP2 replays normally and writes RPKG1 to the local disk, and then sends a notification message to EP 1.
EP1 received the notification message of EP2, since RPKG1 has written most of the databases (EP1 and EP2), advancing C _ SEQNO to 3000 and C _ LSN to 8000.
In the process of replaying the log packet (RPKG1), EP3 may have a data page cache shortage (LSNs of data pages to be modified recorded in the log packet RPKG1 are all larger than C _ LSN recorded in the header of RPKG1, and the data page cannot eliminate the disk-flushing), and EP3 cannot allocate new data pages for data page modification of the replay master library.
EP2 failed, EP3 was in a replay stuck state and could no longer assist master library EP1 in advancing C _ SEQNO and C _ LSN. And the main library also normally receives new adding and deleting modification requests of users, because the C _ LSN of the main library is not advanced, the allocated data page cache of the main library cannot be eliminated, and finally the data page cache of the main library is exhausted, so that the external service cannot be normally provided.
In order to solve the above problems, the method provided by the embodiment of the present invention can avoid the problems of insufficient data page cache, replay jam, and the like in a small number of backup databases, and ensure that the databases can normally and stably operate.
Example one
Fig. 1 is a flowchart of a data page disk-flushing method according to an embodiment of the present invention, where this embodiment is applicable to a case of flushing a data page in a database cluster, and the method may be executed by a data page disk-flushing device, which may be composed of hardware and/or software and may be generally integrated in a device having a data page disk-flushing function.
It should be noted that the method for flushing a data page provided in this embodiment is applied to a database cluster, where the database cluster includes a main library and at least two standby libraries, and the number of the standby libraries is an even number. A database cluster can be understood as a virtual single logical image of a database consisting of a plurality of database servers, which can provide transparent data service for clients and can keep data synchronization in each database in the database cluster. The master database can be understood as a database which allows data modification operations such as addition, deletion, modification and the like in the database cluster and correspondingly generates redo logs; the backup database can be understood as a database which only allows read-only operation in the database cluster, receives the redo log synchronized by the main database, and then performs replay according to the redo log so as to keep the data and the data in the main database synchronized.
As shown in fig. 1, the method for flushing a data page provided in this embodiment includes the following steps:
in step 110, the master library sends the heartbeat message or the generated log packet to each backup library.
The heartbeat message or the log packet carries the first committed log sequence. A committed log sequence may be understood as the largest log sequence among the log sequences that have been written to the majority of the databases in the database cluster (the proportion of databases, including the master, being greater than a preset proportion threshold). Wherein the preset proportion threshold value may be any value greater than 50%.
In this embodiment, the master library may periodically send heartbeat messages to all the standby libraries during normal operation, and when a log packet is sent, the log packet may replace the heartbeat message, and the standby libraries, regardless of whether receiving the heartbeat message of the master library or the log packet of the master library, may update the time of receiving the message of the master library which is stored locally last time, so as to ensure that the timeout condition does not occur. And when the main library sends heartbeat messages or log packets each time, adding the C _ SEQNO and the C _ LSN which are currently and newly pushed into the messages, after the standby library receives the messages, updating the C _ SEQNO and the C _ LSN into the H _ SEQNO and H _ LSN fields which are locally maintained, and still pushing the C _ SEQNO and the C _ LSN of the standby library forward only along with the replay progress of the log packets.
Wherein, the log packet includes a log packet sequence number, a submitted log sequence and at least one log, and may further include: log packet length, minimum log sequence, maximum log sequence, and current any number.
Specifically, the master library may generate the log packet in the following manner: and the master library generates the log packets according to the synchronized log packet sequence number array, the synchronized log sequence array and the received data modification operation.
The synchronized log packet sequence number array can be understood as an array formed by log packet sequence numbers of databases exceeding a set proportion in a database cluster, wherein each synchronized log packet sequence number represents a log packet sequence number of a latest synchronized log packet in a standby database or a master database. The synchronized log sequence array may be understood as an array consisting of the maximum log sequence values of the database in the database cluster exceeding a set proportion of completed log synchronization, wherein each synchronized log sequence value represents the maximum log sequence value of the most recently synchronized log packets in one backup repository or the master repository. Each data modification is identified by using a new Log Sequence Number (LSN), the value range of the LSN is from 0 to infinity, and one LSN value represents one database modification operation.
In this embodiment, the log packet may be understood as including one or more logs generated by data modification performed by the primary library, and needs to be sent to the standby library, so that the standby library performs replay according to the logs therein to implement data synchronization between the primary and secondary libraries. The log packet sequence number can be understood as a number corresponding to the log packet according to the generation sequence, and it can be clear that the log packet sequence number has the characteristic of unique increment, that is, the log packet sequence number of each log packet is uniquely determined, and according to the difference of the generation sequence, the log packet sequence number corresponding to the log packet generated later is inevitably greater than the log packet sequence number of the log packet generated before the log packet. The length of the log packet can be determined according to the size of the disk space occupied by the log packet. The minimum log sequence value can be understood as the smallest log sequence value among the log sequence values corresponding to the logs in the log packet. The maximum log sequence value can be understood as the maximum log sequence value in the log sequence values corresponding to the logs in the log packet. While the current option number may be understood to be the option number of the currently selected master library based on the RAFT protocol, it is understood that the option number is a continuously increasing value, and each time a new master library is selected in the database cluster, the option number of the new master library is larger than the option number of the original master library and is a unique value. The committed log packet sequence number may be understood as the largest log packet sequence number among the log packet sequence numbers that have been written to the majority of the databases in the database cluster (the proportion of databases including the master database is greater than a preset proportion threshold). The log can be understood as a redo log generated by the main library during data modification and requiring the backup library to perform replay according to the redo log to realize data synchronization.
Specifically, the process of the master library generating the log packet according to the synchronized log packet sequence number array, the synchronized log sequence array, and the received data modification operation may be: determining the serial number of the submitted log packet according to the serial number array of the synchronized log packet; determining a submitted log sequence according to the synchronized log sequence group; generating at least one log according to the received data modification operation, and modifying a data page according to the data modification operation; wherein, the page log sequence of the modified data page is the log sequence of the log; adding 1 to the sequence number of the last generated log packet to determine the sequence number of the log packet of the current log packet; and generating a log packet according to the log packet sequence number, the submitted log sequence and at least one log.
The method for determining the submitted log packet sequence number according to the synchronized log packet sequence number array may be that the largest log packet sequence number in the log packet sequence numbers of the synchronized log packet sequence number array, which is greater than or equal to the log packet sequence numbers of the plurality of databases (the ratio of the database ratio including the master database is greater than a preset ratio threshold), is determined as the submitted log packet sequence number. The manner of determining the committed log sequence from the synchronized log sequence set may be: and determining the largest log sequence in the log sequences which are greater than or equal to most of the databases in the synchronized log sequence group (the proportion of the databases including the master database is greater than a preset proportion threshold) as the committed log sequence.
Specifically, when receiving a data modification operation, the master library generates one or more logs to be synchronized, and modifies data in one or more data pages. Wherein, a log is generated every time modification is performed, and the data pages modified by different logs can be the same or different. And the master database determines the latest synchronization-completed log packet sequence number in each database in the database cluster according to the maintained log packet sequence number array, and determines the maximum synchronization-completed log packet sequence number of most databases in the database cluster according to the latest synchronization-completed log packet sequence number. And the main database also determines the maximum log sequence value in the log packet which completes synchronization latest in each database in the database cluster according to the maintained synchronized log sequence value array, and determines the maximum log sequence value which completes synchronization of most databases in the database cluster according to the maximum log sequence value. And generating one or more generated logs, generating log packets according to the information determined by the sequence number array of the synchronized log packets and the sequence value array of the synchronized log packets and the characteristics of the data packets, and sending the log packets to all databases except the main database in the database cluster, namely generating the log packets by the main database in the database cluster and sending the log packets to all standby databases.
Step 120, the backup library extracts a second submitted log sequence in the current replayed log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be flushed.
And the standby database replays the received log packets in sequence. In this embodiment, if the replay progress of the standby database is slow, the currently replayed log packet and the currently received log packet may not be the same log packet.
The second committed log sequence may be understood as a committed log sequence carried in a log packet currently replayed normally by the standby repository. The last log packet written to the local disk may be understood as a log packet newly written to the local disk, and the maximum log sequence may be understood as a corresponding maximum sequence in the logs included in the log packet. The page log sequence in the data page to be flushed may be the sequence of logs generated when the data page is modified.
Step 130, the standby library judges whether the data page to be flushed meets the flushing condition according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence; if the condition for brushing the disc is satisfied, step 140 is executed.
In this embodiment, a series of operations are performed on the first submitted log sequence, the second submitted log sequence, and the maximum log sequence, the operation result is compared with the page log sequence, and whether the data page to be flushed satisfies the flushing condition is determined according to the comparison result.
Specifically, the manner of the standby library judging whether the data page to be flushed satisfies the flushing condition according to the first committed log sequence, the second committed log sequence, the maximum log sequence, and the page log sequence may be: taking the maximum value of the first submitted log sequence and the second submitted log sequence, and determining the maximum value as an intermediate value; taking the minimum value of the intermediate value and the maximum log sequence, and determining the minimum value as a result value; and judging whether the data page to be flushed meets the flushing condition or not according to the result value and the page log sequence.
In this embodiment, if the page log sequence is less than or equal to the result value, the data page to be flushed satisfies the flushing condition; otherwise, the data page to be brushed does not meet the brushing condition. Wherein, the formula satisfying the condition of brushing the disc can be expressed as: page log sequence ≦ min (maximum log sequence, max (first committed log sequence, second committed log sequence)).
In this embodiment, the first committed log sequence may be greater than the second committed log sequence, but for the database cluster, the log corresponding to the first committed log sequence has been written into most databases in the database cluster (the proportion of the databases including the master library is greater than the preset proportion threshold), so even if the replay progress of the current backup library is slow, the data page whose page log sequence is less than or equal to the first committed log sequence may be flushed, because the log less than or equal to the first committed log sequence must not be truncated, but it is also necessary to confirm that the backup library has been written into the maximum log sequence in the last log packet of the local disk, and it is ensured that the log corresponding to the data page modification has been written into the local disk.
According to the scheme, the data page is flushed according to the first submitted log sequence which is actually pushed by the main library, so that more data pages can meet the flushing condition, more free space can be released in time for allocating new data page cache, and the condition that the data page cache is insufficient in a small number of spare libraries can be reduced at a large probability.
And 140, writing the data in the data page to be flushed into the local disk.
According to the technical scheme of the embodiment, the main library sends heartbeat messages or generated log packets to each standby library; the heartbeat message or the log packet carries a first submitted log sequence; the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed; the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence; and if the disk refreshing condition is met, writing the data in the data page to be refreshed into the local disk. According to the data page disk-refreshing method provided by the embodiment of the invention, whether the data page to be refreshed meets the disk-refreshing condition is judged according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence, more memory space can be released in time to allocate new data page cache, the problems of insufficient data page cache and replay blockage of a standby database can be avoided, and the database cluster can be ensured to normally and stably run.
Example two
Fig. 2 is a schematic structural diagram of a data page disk-flushing device according to a second embodiment of the present invention, which is disposed in a database cluster, where the database cluster includes a main library and at least two standby libraries; the device comprises:
a heartbeat message or log packet sending module 210, disposed in the master library, for sending the heartbeat message or the generated log packet to each backup library; the heartbeat message or the log packet carries a first submitted log sequence;
a log sequence obtaining module 220, disposed in the backup repository, configured to extract a second submitted log sequence in a currently replayed log packet and a maximum log sequence in a last log packet written in the local disk, and obtain a page log sequence in a current data page to be flushed;
the disk-brushing condition judging module 230 is arranged in the standby library and is used for judging whether the data page to be brushed meets the disk-brushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence;
and the data page disk-refreshing module 240 is configured to write the data in the data page to be refreshed into the local disk when the disk-refreshing condition is met.
Optionally, the disk-brushing condition determining module 230 is further configured to:
taking the maximum value of the first submitted log sequence and the second submitted log sequence, and determining the maximum value as an intermediate value;
taking the minimum value of the intermediate value and the maximum log sequence, and determining the minimum value as a result value;
and judging whether the data page to be flushed meets the flushing condition or not according to the result value and the page log sequence.
Optionally, the disk-brushing condition determining module 230 includes:
if the page log sequence is less than or equal to the result value, the data page to be flushed meets a flushing condition; otherwise, the data page to be brushed does not meet the brushing condition.
Optionally, the method further includes: a log packet generation module to:
and the master library generates the log packets according to the synchronized log packet sequence number array, the synchronized log sequence array and the received data modification operation.
Optionally, the log packet at least includes a log packet sequence number, a submitted log sequence, and at least one log.
Optionally, the log packet generating module is further configured to:
determining the serial number of the submitted log packet according to the serial number array of the synchronized log packet;
determining a submitted log sequence according to the synchronized log sequence group;
generating at least one log according to the received data modification operation, and modifying a data page according to the data modification operation; wherein, the page log sequence of the modified data page is the log sequence of the log;
adding 1 to the sequence number of the last generated log packet to determine the sequence number of the log packet of the current log packet;
and generating a log packet according to the log packet sequence number, the submitted log sequence and at least one log.
Optionally, data synchronization between the master library and each backup library.
The device can execute the methods provided by all the embodiments of the invention, and has corresponding functional modules and beneficial effects for executing the methods. For details not described in detail in this embodiment, reference may be made to the methods provided in all the foregoing embodiments of the present invention.
EXAMPLE III
Fig. 3 is a schematic structural diagram of an electronic device according to a third embodiment of the present invention. The apparatus comprises: a database cluster 30, a processor 31, a storage 32, an input 33, and an output 34. The database cluster 30 in the electronic device includes a main library 301 and at least two standby libraries 302, the number of the standby libraries 302 is even, and fig. 3 takes two standby libraries 302 as an example. The number of the processors 31 in the electronic device may be one or more, and one processor 31 is taken as an example in fig. 3. The number of the storage devices 32 in the electronic apparatus may be one or more, and one storage device 32 is taken as an example in fig. 3. The database cluster 30, the processor 31, the storage device 32, the input device 33, and the output device 34 of the electronic apparatus may be connected by a bus or other means, and fig. 3 illustrates the connection by the bus as an example.
In this embodiment, the electronic device may be a computer, a notebook, an intelligent tablet, or the like.
The storage device 32 is a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the electronic devices according to any embodiment of the present application. The storage device 32 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the storage device 32 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 32 may further include memory located remotely from the processor 31, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The input means 33 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function controls of the presentation apparatus, and may also be a camera for acquiring images and a sound pickup apparatus for acquiring audio data. The output device 34 may include an audio device such as a speaker. It should be noted that the specific composition of the input device 33 and the output device 34 can be set according to actual conditions.
The processor 31 executes various functional applications of the device and data processing, i.e. implements the above-mentioned method of flushing a data page, by running software programs, instructions and modules stored in the storage means 32.
The electronic device provided by the above can be used to execute the method for flushing the data page provided by any of the above embodiments, and has corresponding functions and advantages.
Example four
Embodiments of the present invention provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processing apparatus, implements a method for flushing a data page according to an embodiment of the present invention.
The computer readable medium of the present invention described above may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: the master library sends the heartbeat message or the generated log packet to each standby library; wherein the heartbeat message or the log packet carries a first submitted log sequence; the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed; the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence; and if the disk refreshing condition is met, writing the data in the data page to be refreshed into a local disk.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data page disk-refreshing method is applied to a database cluster, wherein the database cluster comprises a main library and at least two standby libraries; the method comprises the following steps:
the master library sends heartbeat messages or generated log packets to the standby libraries; wherein the heartbeat message or the log packet carries a first submitted log sequence;
the standby library extracts a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk, and obtains a page log sequence in the current data page to be refreshed;
the standby library judges whether the data page to be flushed meets the flushing condition or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence;
and if the disk refreshing condition is met, writing the data in the data page to be refreshed into a local disk.
2. The method of claim 1, wherein the step of the standby library determining whether the data page to be flushed satisfies a flushing condition according to the first committed log sequence, the second committed log sequence, the maximum log sequence, and the page log sequence comprises:
taking the maximum value of the first submitted log sequence and the second submitted log sequence, and determining the maximum value as an intermediate value;
taking the intermediate value and the minimum value of the maximum log sequence, and determining the intermediate value and the minimum value as a result value;
and judging whether the data page to be flushed meets a flushing condition or not according to the result value and the page log sequence.
3. The method according to claim 2, wherein determining whether the data page to be flushed satisfies a flushing condition according to the result value and the page log sequence comprises:
if the page log sequence is less than or equal to the result value, the data page to be flushed meets a flushing condition; otherwise, the data page to be brushed does not meet the brushing condition.
4. The method of claim 1, wherein the master library generates a log package comprising:
and the master library generates log packets according to the synchronized log packet sequence number array, the synchronized log sequence array and the received data modification operation.
5. The method of claim 4, wherein the log packets comprise at least a log packet sequence number, a committed log sequence, and at least one log.
6. The method of claim 5, wherein generating log packets by the master library based on the synchronized log packet sequence number array, the synchronized log sequence array, and the received data modification operation comprises:
determining the serial number of the submitted log packet according to the serial number array of the synchronized log packet;
determining a submitted log sequence according to the synchronized log sequence group;
generating at least one log according to the received data modification operation, and modifying a data page according to the data modification operation; wherein the page log sequence of the modified data page is the log sequence of the log;
adding 1 to the sequence number of the last generated log packet to determine the sequence number of the log packet of the current log packet;
and generating a log packet according to the log packet sequence number, the submitted log sequence and the at least one log.
7. The method of any of claims 1-6, wherein data is synchronized between the master library and each of the backup libraries.
8. A data page disk brushing device is characterized by being arranged in a database cluster, wherein the database cluster comprises a main library and at least two standby libraries; the device comprises:
the heartbeat message or log packet sending module is arranged in the main library and is used for sending the heartbeat message or the generated log packet to each standby library; wherein the heartbeat message or the log packet carries a first submitted log sequence;
the log sequence acquisition module is arranged in the standby library and used for extracting a second submitted log sequence in the current replay log packet and a maximum log sequence in the last log packet written into the local disk and acquiring a page log sequence in a current data page to be refreshed;
the disk-brushing condition judging module is arranged in the standby library and used for judging whether the data pages to be brushed meet the disk-brushing conditions or not according to the first submitted log sequence, the second submitted log sequence, the maximum log sequence and the page log sequence;
and the data page disk-brushing module is used for writing the data in the data page to be brushed into the local disk when the disk-brushing condition is met.
9. An electronic device, comprising: a database cluster, storage, and one or more processors;
the database cluster comprises a main library and at least two standby libraries;
the storage device to store one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of flushing a data page of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out a method of flushing a data page according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110901940.8A CN113553382B (en) | 2021-08-06 | 2021-08-06 | Method, device, equipment and storage medium for brushing data pages |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110901940.8A CN113553382B (en) | 2021-08-06 | 2021-08-06 | Method, device, equipment and storage medium for brushing data pages |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113553382A true CN113553382A (en) | 2021-10-26 |
| CN113553382B CN113553382B (en) | 2023-06-02 |
Family
ID=78105436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110901940.8A Active CN113553382B (en) | 2021-08-06 | 2021-08-06 | Method, device, equipment and storage medium for brushing data pages |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113553382B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239476A (en) * | 2014-09-04 | 2014-12-24 | 上海天脉聚源文化传媒有限公司 | Method, device and system for synchronizing databases |
| CN105975579A (en) * | 2016-05-05 | 2016-09-28 | 北京思特奇信息技术股份有限公司 | Primary and backup copying method of memory database and memory database system |
| CN106933843A (en) * | 2015-12-29 | 2017-07-07 | 阿里巴巴集团控股有限公司 | database heartbeat detecting method and device |
| CN108376142A (en) * | 2018-01-10 | 2018-08-07 | 北京思特奇信息技术股份有限公司 | A kind of distributed memory database method of data synchronization and system |
| US20180246911A1 (en) * | 2017-02-28 | 2018-08-30 | Sap Se | Database Memory Management in a High Availability Database System Using Limits |
| CN109783200A (en) * | 2017-11-13 | 2019-05-21 | 阿里巴巴集团控股有限公司 | Data processing method, device and equipment |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN110019066A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data base processing method and device, system |
| CN110543386A (en) * | 2019-09-16 | 2019-12-06 | 上海达梦数据库有限公司 | Data storage method, device, equipment and storage medium |
| CN112596953A (en) * | 2020-12-25 | 2021-04-02 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
-
2021
- 2021-08-06 CN CN202110901940.8A patent/CN113553382B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104239476A (en) * | 2014-09-04 | 2014-12-24 | 上海天脉聚源文化传媒有限公司 | Method, device and system for synchronizing databases |
| CN106933843A (en) * | 2015-12-29 | 2017-07-07 | 阿里巴巴集团控股有限公司 | database heartbeat detecting method and device |
| CN105975579A (en) * | 2016-05-05 | 2016-09-28 | 北京思特奇信息技术股份有限公司 | Primary and backup copying method of memory database and memory database system |
| US20180246911A1 (en) * | 2017-02-28 | 2018-08-30 | Sap Se | Database Memory Management in a High Availability Database System Using Limits |
| CN110019062A (en) * | 2017-08-14 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method of data synchronization and system |
| CN110019066A (en) * | 2017-09-21 | 2019-07-16 | 阿里巴巴集团控股有限公司 | Data base processing method and device, system |
| CN109783200A (en) * | 2017-11-13 | 2019-05-21 | 阿里巴巴集团控股有限公司 | Data processing method, device and equipment |
| CN108376142A (en) * | 2018-01-10 | 2018-08-07 | 北京思特奇信息技术股份有限公司 | A kind of distributed memory database method of data synchronization and system |
| CN110543386A (en) * | 2019-09-16 | 2019-12-06 | 上海达梦数据库有限公司 | Data storage method, device, equipment and storage medium |
| CN112596953A (en) * | 2020-12-25 | 2021-04-02 | 上海达梦数据库有限公司 | Database log replay method, device, equipment and storage medium |
Non-Patent Citations (5)
| Title |
|---|
| 刘JAVA: "MySQL数据和日志的刷盘机制以及双一配置", Retrieved from the Internet <URL:https://blog.csdn.net/weixin_43767015/article/details/118549790> * |
| 唐丽园: "面向分布式系统的数据同步策略研究", 《2019航空装备服务保障与维修技术论坛暨中国航空工业技术装备工程协会年会论文集》 * |
| 大将黄猿: "InnoDB的数据页刷盘时机", Retrieved from the Internet <URL:https://blog.csdn.net/weixin_47184173/article/details/117433439> * |
| 湘水友独: "InnoDB存储引擎——redo log", Retrieved from the Internet <URL:https://blog.csdn.net/czxlylc/article/details/113196876> * |
| 王蒙蒙等: "SQL Server数据库的应用级持续数据保护系统", 《计算机系统应用》, vol. 24, no. 2 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113553382B (en) | 2023-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12019652B2 (en) | Method and device for synchronizing node data | |
| CN108280080B (en) | Data synchronization method and device and electronic equipment | |
| US11500879B2 (en) | Method, device, and program product for managing index of streaming data storage system | |
| US11586388B2 (en) | Method, device, and computer program product for managing storage system | |
| CN115517009B (en) | Cluster management method, cluster management device, storage medium and electronic equipment | |
| CN108369547B (en) | System and method for managing tails of logs in persistent main memory | |
| CN112416654B (en) | Database log replay method, device, equipment and storage medium | |
| CN111124270A (en) | Method, apparatus and computer program product for cache management | |
| US11599290B2 (en) | Data storage method, electronic device, and computer program product | |
| CN111444147A (en) | Data page creating method and device, terminal equipment and storage medium | |
| CN110121694B (en) | Log management method, server and database system | |
| US10938633B2 (en) | Facilitating resilient and fault tolerant asynchronous messaging | |
| Bouteiller et al. | Reasons for a pessimistic or optimistic message logging protocol in MPI uncoordinated failure, recovery | |
| US11567814B2 (en) | Message stream processor microbatching | |
| CN115061630A (en) | Data migration method, device, equipment and medium | |
| US11740827B2 (en) | Method, electronic device, and computer program product for recovering data | |
| CN112948410A (en) | Data processing method, device, equipment and medium | |
| CN113553382A (en) | Data page disk refreshing method, device, equipment and storage medium | |
| CN116527590A (en) | Distributed current limiting implementation method and device for cloud native gateway | |
| US20180309702A1 (en) | Method and device for processing data after restart of node | |
| US10819622B2 (en) | Batch checkpointing for inter-stream messaging system | |
| CN113297327A (en) | System and method for generating distributed ID | |
| CN113760862A (en) | Incremental data breakpoint continuous transmission method, device, equipment and storage medium | |
| CN108984431B (en) | Method and apparatus for flushing stale caches | |
| CN111026764A (en) | Data storage method and device, electronic product and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |