CN110019498B - Log synchronization method and device, storage medium and electronic equipment - Google Patents
Log synchronization method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN110019498B CN110019498B CN201710692113.6A CN201710692113A CN110019498B CN 110019498 B CN110019498 B CN 110019498B CN 201710692113 A CN201710692113 A CN 201710692113A CN 110019498 B CN110019498 B CN 110019498B
- Authority
- CN
- China
- Prior art keywords
- log
- transaction
- queue
- reading
- transaction logs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The disclosure relates to a log synchronization method and device, and belongs to the technical field of data processing. The method comprises the following steps: reading a transaction log from a source database and storing the transaction log into a plurality of read ring queues; storing the transaction logs in each reading circular queue into a consumption circular queue, and sending the transaction logs in the consumption circular queue to a thread which is connected with a target database; and the thread writes the transaction log into the target database and records the log code corresponding to the successfully written transaction log. The method has the advantages that the transaction log is stored by adopting the annular queue, the transaction log storage speed is increased, and the memory loss is greatly reduced.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a log synchronization method, a log synchronization apparatus, a computer-readable storage medium, and an electronic device.
Background
With the development of internet technology, more and more users using the internet are provided, and in order to ensure the stability of the system, the deployment structure of the system is also deployed from the initial single machine room, and gradually changes to the deployment of multiple machine rooms in the same city, multiple machine rooms in different places and the like. When a system is deployed in multiple machine rooms, generated data can be distributed in the multiple machine rooms, so that great challenges are brought to data synchronization among the multiple machine rooms.
The current methods for maintaining synchronization between data may include the following two methods: one is to rely on the synchronization mechanism provided by the data source software to complete data synchronization; for example, MySql (relational database management system) or Redis, etc. may provide a corresponding mechanism to accomplish data synchronization; the other is to ensure data consistency by applying double writing; the method specifically comprises the following steps: after completing the write operation to data source A, the write operation is performed to data source B at the same time.
However, both of the above methods have corresponding disadvantages: firstly, for MySql, depending on a synchronization mechanism provided by the MySql, when the load pressure of a source database is large, data synchronization delay can be caused; for Redis, when Redis data synchronization is used, when a master node is replaced or the master node log buffer range is exceeded, the situation of full data synchronization occurs, so that the data synchronization time is prolonged, and Redis cannot provide services to the outside during the data synchronization; secondly, double writing is applied to invade the data consistency logic into the application program, so that the program logic is relatively complex; and the writing operation is carried out on a plurality of data sources at the same time, and when the condition that partial data sources fail to be written occurs, the consistency of all data sources is difficult.
Therefore, it is necessary to provide a new log synchronization method.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
An object of the present disclosure is to provide a log synchronization method, a log synchronization apparatus, a computer-readable storage medium, and an electronic device, which overcome one or more of the problems due to the limitations and disadvantages of the related art, at least to some extent.
According to an aspect of the present disclosure, there is provided a log synchronization method, including:
reading a transaction log from a source database and storing the transaction log into a plurality of read ring queues;
storing the transaction logs in each reading circular queue into a consumption circular queue, and sending the transaction logs in the consumption circular queue to a thread which is connected with a target database;
and the thread writes the transaction log into the target database and records the log code corresponding to the successfully written transaction log.
In an exemplary embodiment of the present disclosure, the transaction log includes a plurality of data sources, and each of the data sources corresponds to each of the read ring queues in a one-to-one manner.
In an exemplary embodiment of the present disclosure, after storing the transaction log in a plurality of read ring queues, the log synchronization method further includes:
and encoding the transaction logs in each read circular queue in an incremental mode.
In an exemplary embodiment of the present disclosure, after storing the transaction log in a plurality of read ring queues, the log synchronization method further includes:
and storing the transaction logs in the reading circular queue to a local database.
In an exemplary embodiment of the present disclosure, before storing the transaction log in a plurality of read ring queues, the log synchronization method further includes:
and analyzing the transaction log, and formatting the analyzed transaction log.
In an exemplary embodiment of the present disclosure, storing the transaction log in each of the read ring queues into a consumption ring queue includes:
sending the log code corresponding to the transaction log in each reading circular queue to the consumption circular queue;
judging whether the transaction logs corresponding to the log codes exist in the reading circular queue or not;
when the transaction logs corresponding to the log codes are judged to exist in the reading annular queue, storing the transaction logs corresponding to the log codes into a consumption annular queue;
and when the transaction logs corresponding to the log codes do not exist in the reading annular queue, acquiring the transaction logs corresponding to the log codes from the local database and storing the transaction logs into a consumption annular queue.
In an exemplary embodiment of the disclosure, sending the transaction log in the consumption ring queue to a thread that establishes a connection with a target database includes:
starting a thread which establishes connection with the target database and taking out the transaction log from the consumption circular queue;
and generating an execution task from the transaction log and sending the execution task to the thread.
In an exemplary embodiment of the present disclosure, before reading the transaction log from the source database, the log synchronization method further includes:
and starting a thread which establishes connection with the source database and sending a reading request for reading the transaction log to the source database.
According to an aspect of the present disclosure, there is provided a log synchronizing apparatus including:
the reading module is used for reading the transaction logs from the source database and storing the transaction logs into a plurality of reading ring queues;
the sending module is used for storing the transaction logs in each reading annular queue into a consumption annular queue and sending the transaction logs in the consumption annular queue to a thread which is connected with a target database;
and the writing module is used for writing the transaction log into the target database by the thread and recording the log code corresponding to the successfully written transaction log.
According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the log synchronization method of any one of the above.
According to an aspect of the present disclosure, there is provided an electronic device including:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform any of the log synchronization methods described above via execution of the executable instructions.
The invention discloses a log synchronization method and a log synchronization device, wherein a transaction log is read from a source database and stored in a circular queue; then, the transaction logs in the ring queue are sent to a thread which is connected with a target database; finally, the thread writes the transaction log into a target database and records the log code corresponding to the successfully written transaction log; on one hand, by recording the log code corresponding to the transaction log which is successfully written in, the transaction log corresponding to the synchronized transaction log can be synchronized according to the log code without writing the transaction log corresponding to the synchronized log code again, so that the transaction logs can be synchronized in an incremental mode, the full synchronization of the transaction logs is avoided, the synchronization efficiency of the transaction logs is improved, and the load of target data is reduced; on the other hand, the transaction log is stored by adopting the annular queue, so that the transaction log storage speed is improved, and the memory loss is greatly reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
Fig. 1 schematically shows a flow chart of a log synchronization method.
Fig. 2 schematically shows an initial state diagram of a circular queue.
Fig. 3 is a schematic diagram showing a state after adding a data to a circular queue.
FIG. 4 is a diagram schematically illustrating a status of a circular queue after read and add.
FIG. 5 schematically illustrates a flow diagram of transaction log acquisition.
FIG. 6 schematically illustrates a flow chart of a method of storing transaction logs in a consumption ring queue.
FIG. 7 schematically illustrates a flow diagram of transaction log synchronization.
Fig. 8 schematically shows a block diagram of a log synchronization apparatus.
Fig. 9 schematically shows an exemplary diagram of an electronic device for implementing the log synchronization method described above.
Fig. 10 schematically illustrates a computer storage medium for implementing the log synchronization method described above.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The exemplary embodiment first provides a log synchronization method. Referring to fig. 1, the log synchronization method may include the steps of:
step S110, reading a transaction log from a source database and storing the transaction log into a plurality of reading ring queues.
Step S120, storing the transaction logs in each reading annular queue into a consumption annular queue, and sending the transaction logs in the consumption annular queue to a thread which is connected with a target database.
And S130, writing the transaction log into the target database by the thread and recording a log code corresponding to the successfully written transaction log.
In the log synchronization method, on one hand, by recording the log code corresponding to the successfully written transaction log, when synchronization is performed again next time, synchronization can be performed according to the log code without writing the transaction log corresponding to the synchronized transaction log code again, so that the transaction logs can be synchronized in an incremental manner, full synchronization of the transaction logs is avoided, the synchronization efficiency of the transaction logs is improved, and the load of target data is reduced; on the other hand, the transaction log is stored by adopting the annular queue, so that the transaction log storage speed is improved, and the memory loss is greatly reduced.
Next, each step in the above-described log synchronizing method in the present exemplary embodiment will be explained and explained in detail.
In step S110, a transaction log is read from a source database and stored in a plurality of read ring queues.
First, the source database and the transaction log are explained and explained. Wherein, the Source Database (Source Database/Source Databases) is a Database directly providing original information data or specific data; the source database may include a full-text database, a term database, a numerical database, a text numerical database, an image database, and the like, and may also include other types of databases, such as an audio database or a video database, and the like, which is not limited in this disclosure; the Transaction Log (Transaction Log) extension may include ldf, a file that may be used to record updates to the source database; any operation on the database can be recorded, and the recorded result is saved in an independent file; in addition, for each database updating process, the transaction log has very comprehensive records, and the state before the database updating can be recovered according to the records; further, the transaction log may include a plurality of data sources, which may include, for example, observation data sources, analytical measurement data sources, graphic data sources, statistical survey data sources, telemetry data sources, and the like.
Next, the above-described circular queue is explained and explained. A Ring Buffer (also referred to as a Ring Buffer) may be used to store data transmitted and received in a communication. The circular queue is a first-in-first-out circular buffer that provides the communication program with mutually exclusive access to the buffer. Further, the ring buffer may generally include a read pointer and a write pointer; wherein the read pointer points to data readable in the ring buffer and the write pointer points to a writable buffer in the ring buffer. The data reading and writing of the buffer area can be realized by moving the reading pointer and the writing pointer; in the usual case, the read user of the ring buffer will influence the read pointer tightly, while the write user will only influence the write pointer. If only one reading user and one writing user exist, the correctness of the data can be ensured without adding a mutual exclusion protection mechanism. If a plurality of read-write users access the ring buffer, a mutual exclusion protection mechanism can be added to ensure that the plurality of users access the ring buffer mutually exclusively. The working process of the circular queue can be shown by referring to fig. 2-4:
where FIG. 2 is the initial state of the ring buffer, it can be seen that both the read pointer and the write pointer point to the first buffer; FIG. 3 is a situation after adding a data to the ring buffer, and it can be seen that the write pointer has moved to the location of data block 2, while the read pointer has not moved; fig. 4 shows the ring buffer after reading and adding, and it can be seen that two data have been added to the ring buffer and one data has been read.
Finally, the above step S110 is exemplified based on the explanation and description of the source database, the transaction log and the circular queue. For example:
referring to fig. 5, first, a transaction log is read from a Source Database (Source Database), and then the read transaction log is stored in a read Ring queue (Ring Buffer). It should be further added that, since the transaction log may include a plurality of data sources, the read circular queue also includes a plurality of data sources, and the number of the read circular queue is in one-to-one correspondence with the number of the data sources; after a transaction log including a plurality of data sources is read, whether a read circular queue associated with each data source included in the read transaction log exists in an original read circular queue or not can be judged, and if the read circular queue exists, the read data source is directly written into the read circular queue associated with the data source; if the read data source does not exist in the read ring queue, a read ring queue can be newly built, and then the read data source is written into the newly built read ring queue.
Further, in order to successfully read the transaction log from the source database, a connection with the source database needs to be established, which may specifically include: and starting a thread which establishes connection with the source database and sending a reading request for reading the transaction log to the source database. In detail:
referring to fig. 5, a thread that establishes a connection with a source database is started to establish a connection with the source database; then sending a reading request for reading the transaction log to a source database; and after the source database responds to the reading request, reading the transaction log in the source database. It should be noted here that, in order to increase the reading speed, a plurality of threads may be established, so that the number of each data source corresponds to the number of each thread one by one, that is, one thread corresponds to one data source; in order to avoid the problem that a system is overloaded due to the establishment of a plurality of threads, one thread can correspond to a plurality of data sources; the present disclosure is not limited thereto.
Furthermore, in order to save the storage space of each read circular queue, the read transaction log may be preprocessed, which specifically includes: and analyzing the transaction log, and formatting the analyzed transaction log. In detail:
after reading a transaction log comprising a plurality of data sources from a source database, firstly analyzing the transaction log, and analyzing the transaction logs in different formats into transaction logs in the same format (for example, all the transaction logs in the same format) so as to compare the transaction logs of the data sources at a later stage, and the like; and then formatting the analyzed transaction log so as to save the storage space for reading the annular queue, reduce the overall burden of the system and improve the reading efficiency of the transaction log.
Still further, in order to prevent the situation that the transaction logs synchronized last time are repeatedly synchronized to cause full synchronization in the process of disconnecting the target database and then connecting the target database to the network and performing synchronization again, the transaction logs in each read ring queue may be encoded, which specifically includes: and encoding the transaction logs in each read circular queue in an incremental mode. For example:
referring to FIGS. 2-4, after the first module in the read circular queue has been stored, it is marked 1 and then the next module is marked 2; when the transaction log is stored in the read circular queue again next time, 1 (or 2 or other numbers may be added, which is not specifically limited by this disclosure) may be added to the latest tag as the encoding for this time.
Finally, in order to avoid the loss of the transaction logs in the read ring queue due to the slower synchronization speed of the target database, the transaction logs in the read ring queue may be stored in a local database (for example, a local disk or a removable hard disk, etc., which is not limited in this example). In detail:
in order to facilitate the consumption of the circular queue, all the transaction logs read by the reading circular queue from the source database can be stored in the target database, and all the transaction logs read from the source database can be found, and all the transaction logs in the reading circular queue can be stored in the local disk. It should be added that, in the local disk, for convenience of query, the transaction logs corresponding to the data sources may be stored separately.
Further, a storage mode of storing the transaction log in the local database is further described in an additional way. The method specifically comprises the following steps: and storing the transaction log to a local database in a log snapshot mode. In detail:
the first step, creating a Dblink (database connection) between the read ring queue and the local disk; secondly, establishing a synchronous table of transaction logs on the reading ring queue and the local disk; thirdly, testing Dblink on the local disk to ensure the correct connection between the read ring queue and the local disk; fourthly, creating a log to be snapshot on the read annular queue; fifthly, creating a snapshot on a local disk; and sixthly, setting snapshot refreshing time. In addition, in other exemplary embodiments of the present disclosure, the transaction log may also be stored in a local disk in other manners, for example, it may be copied or sent, and the like, which is not limited in this example.
In step S120, the transaction logs in each read ring queue are stored in a consumption ring queue, and the transaction logs in the consumption ring queue are sent to the thread that establishes a connection with the target database.
First, as shown with reference to FIG. 6, storing the transaction log in the consumption circular queue may include steps S610-S640. Wherein:
in step S610, a log code corresponding to the transaction log in each reading circular queue is sent to the consumption circular queue. In detail:
referring to fig. 7, first, a connection thread between the read ring queue and the consumption ring queue is started, and a connection between the read ring queue and the consumption ring queue is established; and then sending the log code corresponding to the transaction log to be synchronized to a consumption circular queue. It should be added that, in order to increase the reading speed, a mode that one thread corresponds to one consumption ring queue may be adopted; meanwhile, in order to avoid the system overload caused by establishing a plurality of threads, one thread may also correspond to a plurality of data sources, which is not limited in this disclosure.
In step S620, it is determined whether a transaction log corresponding to each log code exists in the read ring queue. In detail:
the method has a certain capacity limit due to reading the transaction logs which can be stored in the circular queue; therefore, when the target database synchronization log encoding is far behind the log encoding in the read ring queue, the transaction log may not be stored in the read ring queue and then is transferred to the local disk. Therefore, in order to successfully obtain the transaction logs corresponding to the log codes, it is necessary to first determine whether the transaction logs corresponding to the log codes exist in the read ring queue.
In step S630, when it is determined that the transaction log corresponding to each log code exists in the read ring queue, the transaction log corresponding to each log code is stored in the consumption ring queue. In detail:
when the transaction logs corresponding to the log codes are judged to exist in the reading circular queue, the transaction logs corresponding to the log codes can be obtained and sent to a thread connected with the reading circular queue and the consumption circular queue, and then the transaction logs corresponding to the log codes are stored in the consumption circular queue through the thread.
In step S640, when it is determined that the transaction log corresponding to each log code does not exist in the read ring queue, the transaction log corresponding to each log code is obtained from the local database and stored in the consumption ring queue. In detail:
when the transaction logs corresponding to the log codes are judged not to exist in the reading annular queue, firstly starting threads between the consumption annular queue and the local disk to establish connection between the consumption annular queue and the local disk; and then inquiring transaction logs corresponding to the log codes in the local disk according to the log codes, sending the transaction logs corresponding to the log codes to a thread connected with the local disk of the consumption annular queue, and storing the transaction logs corresponding to the log codes into the consumption annular queue through the thread.
Secondly, after storing the transaction logs corresponding to the log codes into the consumption ring queue, the transaction logs in the consumption ring queue need to be sent to a thread connected with the target database, and the process may include: starting a thread which establishes connection with the target database and taking out the transaction log from the consumption circular queue; and generating an execution task from the transaction log and sending the execution task to the thread. In detail:
firstly, starting a forwarding (Dispatcher) thread, then taking out transaction logs in batch from a consumption circular queue through the thread, and grouping the transaction logs according to data sources to obtain a plurality of transaction log file packets with the data sources as names; generating a plurality of execution tasks by using the transaction log file packet with each data source as a name; each execution task is then sent to a pool of execution threads that includes a plurality of threads (threads). It should be further added that, in order to increase the execution speed, a manner that one thread corresponds to one transaction log file packet with each data source as a name may be adopted; meanwhile, in order to avoid the system overload caused by establishing a plurality of threads, one thread may correspond to a plurality of transaction log file packets named by each data source, which is not limited in this disclosure.
In step S130, the thread writes the transaction log into the target database and records a log code corresponding to the successfully written transaction log.
In this exemplary embodiment, when a thread in the execution thread pool receives a transaction log file packet with each data source as a name, a log statement is executed, and then the transaction log file packet with each data source as a name is submitted; and after the transaction logs are submitted, recording the log codes corresponding to the transaction logs which are successfully submitted. By recording the log codes corresponding to the transaction logs which are successfully written in, the transaction logs corresponding to the synchronized transaction logs can be synchronized according to the log codes without writing the transaction logs corresponding to the synchronized transaction logs again, so that the transaction logs can be synchronized in an incremental mode, the full synchronization of the transaction logs is avoided, the synchronization efficiency of the transaction logs is improved, and the load of target data is reduced.
Moreover, although the steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
The present disclosure also provides a log synchronization device. Referring to fig. 8, the log synchronizing apparatus may include: a read module 810, a send module 820, and a write module 830. Wherein:
the read module 810 may be configured to read a transaction log from a source database and store the transaction log in a plurality of read ring queues.
The sending module 820 may be configured to store the transaction log in each read ring queue into a consumption ring queue, and send the transaction log in the consumption ring queue to a thread that establishes a connection with a target database.
The writing module 830 may be configured to write the transaction log into the target database by the thread and record a log code corresponding to the successfully written transaction log.
The specific details of each module in the log synchronization device have been described in detail in the corresponding log synchronization method, and therefore are not described herein again.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 9. The electronic device 600 shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 9, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: the at least one processing unit 610, the at least one memory unit 620, and a bus 630 that couples the various system components including the memory unit 620 and the processing unit 610.
Wherein the storage unit stores program code that is executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention as described in the above section "exemplary methods" of the present specification. For example, the processing unit 610 may perform step S110 as shown in fig. 2: reading a transaction log from a source database and storing the transaction log into a plurality of read ring queues; s120: storing the transaction logs in each reading circular queue into a consumption circular queue, and sending the transaction logs in the consumption circular queue to a thread which is connected with a target database; step S130: and the thread writes the transaction log into the target database and records the log code corresponding to the successfully written transaction log.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. As shown, the network adapter 660 communicates with the other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
Referring to fig. 10, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (10)
1. A log synchronization method, comprising:
reading a transaction log from a source database and storing the transaction log into a plurality of read ring queues;
storing the transaction logs in each reading annular queue to a consuming annular queue, storing the transaction logs in the reading annular queue to a local database in a log snapshot mode, starting a forwarding thread, and taking out the transaction logs in batch from the consuming annular queue through the forwarding thread;
grouping the transaction logs according to data sources to obtain a plurality of transaction log file packets with the data sources as names;
generating a plurality of execution tasks according to the transaction log file packet with each data source as a name;
sending the execution task to an execution thread pool;
executing log statements included in the execution task, and submitting the transaction log file packet taking each data source as a name to a target database;
and recording the log code corresponding to the transaction log which is successfully submitted.
2. The log synchronization method of claim 1, wherein the transaction log comprises a plurality of data sources, and each data source corresponds to each read ring queue one to one.
3. The log synchronization method of claim 1, wherein after storing the transaction logs in a plurality of read ring queues, the log synchronization method further comprises:
and encoding the transaction logs in each read circular queue in an incremental mode.
4. The log synchronization method of claim 1, wherein prior to storing the transaction logs in a plurality of read ring queues, the log synchronization method further comprises:
and analyzing the transaction log, and formatting the analyzed transaction log.
5. The log synchronization method of claim 1, wherein storing the transaction log in each read ring queue into a consumption ring queue comprises:
sending the log code corresponding to the transaction log in each reading circular queue to the consumption circular queue;
judging whether the transaction logs corresponding to the log codes exist in the reading circular queue or not;
when the transaction logs corresponding to the log codes are judged to exist in the reading annular queue, storing the transaction logs corresponding to the log codes into a consumption annular queue;
and when the transaction logs corresponding to the log codes do not exist in the reading annular queue, acquiring the transaction logs corresponding to the log codes from the local database and storing the transaction logs into a consumption annular queue.
6. The log synchronization method of claim 1, wherein sending the transaction log in the consumption ring queue to a thread that establishes a connection with a target database comprises:
starting a thread which establishes connection with the target database and taking out the transaction log from the consumption circular queue;
and generating an execution task from the transaction log and sending the execution task to the thread.
7. The log synchronization method of claim 1, wherein prior to reading the transaction log from the source database, the log synchronization method further comprises:
and starting a thread which establishes connection with the source database and sending a reading request for reading the transaction log to the source database.
8. A log synchronization apparatus, comprising:
the reading module is used for reading the transaction logs from the source database and storing the transaction logs into a plurality of reading ring queues;
a sending module, configured to store the transaction logs in each read ring queue to a consumption ring queue, store the transaction logs in the read ring queue to a local database in a log snapshot manner, start a forwarding thread, and batch-take out the transaction logs from the consumption ring queue through the forwarding thread;
grouping the transaction logs according to data sources to obtain a plurality of transaction log file packets with the data sources as names;
generating a plurality of execution tasks according to the transaction log file packet with each data source as a name;
sending the execution task to an execution thread pool;
the writing module is used for executing the log statements included in the execution task and submitting the transaction log file packet taking each data source as a name to a target database;
and recording the log code corresponding to the transaction log which is successfully submitted.
9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the log synchronization method of any one of claims 1 to 7.
10. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the log synchronization method of any of claims 1-7 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710692113.6A CN110019498B (en) | 2017-08-14 | 2017-08-14 | Log synchronization method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710692113.6A CN110019498B (en) | 2017-08-14 | 2017-08-14 | Log synchronization method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019498A CN110019498A (en) | 2019-07-16 |
| CN110019498B true CN110019498B (en) | 2022-04-12 |
Family
ID=67186073
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710692113.6A Active CN110019498B (en) | 2017-08-14 | 2017-08-14 | Log synchronization method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019498B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457181B (en) * | 2019-08-02 | 2023-05-16 | 武汉达梦数据库股份有限公司 | Log optimization analysis method and device for database |
| CN110519343B (en) * | 2019-08-13 | 2020-12-04 | 创新奇智(北京)科技有限公司 | Event synchronization method based on log |
| CN111061690B (en) * | 2019-11-22 | 2023-08-22 | 武汉达梦数据库股份有限公司 | RAC-based database log file reading method and device |
| CN110968438B (en) * | 2019-11-29 | 2022-09-02 | 江苏满运软件科技有限公司 | Asynchronous notification method and device of event message, electronic equipment and storage medium |
| CN112306827A (en) * | 2020-03-25 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Log collection device, method and computer readable storage medium |
| CN111858447A (en) * | 2020-07-13 | 2020-10-30 | 深圳市集贤科技有限公司 | Method for receiving data from serial port and processing data frame |
| CN111897867A (en) * | 2020-08-17 | 2020-11-06 | 杭州安恒信息技术股份有限公司 | Database log statistical method, system and related device |
| CN114328563B (en) * | 2021-12-31 | 2023-07-14 | 北京深演智能科技股份有限公司 | Data updating control method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| US20130318044A1 (en) * | 2010-07-27 | 2013-11-28 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| CN103596208A (en) * | 2013-11-15 | 2014-02-19 | 大唐移动通信设备有限公司 | Method and system for judging fault of network element |
| CN104539669A (en) * | 2014-12-17 | 2015-04-22 | 中国电子科技集团公司第十五研究所 | Data synchronization method based on mobile terminal |
| CN105224444A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | Daily record generation method and device |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
| CN106649756A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Log synchronization method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106897206A (en) * | 2015-12-18 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of service test method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101719149B (en) * | 2009-12-03 | 2012-02-08 | 联动优势科技有限公司 | Data synchronization method and device |
-
2017
- 2017-08-14 CN CN201710692113.6A patent/CN110019498B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130318044A1 (en) * | 2010-07-27 | 2013-11-28 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN103596208A (en) * | 2013-11-15 | 2014-02-19 | 大唐移动通信设备有限公司 | Method and system for judging fault of network element |
| CN104539669A (en) * | 2014-12-17 | 2015-04-22 | 中国电子科技集团公司第十五研究所 | Data synchronization method based on mobile terminal |
| CN105224444A (en) * | 2015-10-14 | 2016-01-06 | 深圳市金证科技股份有限公司 | Daily record generation method and device |
| CN105224445A (en) * | 2015-10-28 | 2016-01-06 | 北京汇商融通信息技术有限公司 | Distributed tracking system |
| CN106897206A (en) * | 2015-12-18 | 2017-06-27 | 阿里巴巴集团控股有限公司 | A kind of service test method and device |
| CN106708578A (en) * | 2016-12-23 | 2017-05-24 | 北京五八信息技术有限公司 | Dual-thread-based journal output method and device |
| CN106649756A (en) * | 2016-12-26 | 2017-05-10 | 北京奇虎科技有限公司 | Log synchronization method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019498A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019498B (en) | Log synchronization method and device, storage medium and electronic equipment | |
| CN110806958A (en) | Monitoring method, monitoring device, storage medium and electronic equipment | |
| CN111913867A (en) | Fault feedback method, device, equipment and storage medium | |
| CN110990081A (en) | Microservice registration and discovery method and device, storage medium and electronic equipment | |
| CN113742628A (en) | Form processing method and device, computer readable storage medium and electronic equipment | |
| CN110489158B (en) | Method, device, medium and electronic equipment for optimizing code hosting platform | |
| CN111767296A (en) | Method, device, electronic equipment and readable storage medium for synchronizing data | |
| CN109697034B (en) | Data writing method and device, electronic equipment and storage medium | |
| CN111338834A (en) | Data storage method and device | |
| CN110688145A (en) | Android MVP code automatic generation method, device, medium and electronic equipment | |
| CN110716804A (en) | Method and device for automatically deleting useless resources, storage medium and electronic equipment | |
| CN113778897A (en) | Automatic test method, device, equipment and storage medium of interface | |
| CN109408057B (en) | Method, device, medium and computing equipment for automatically generating codes | |
| CN109067649B (en) | Node processing method and device, storage medium and electronic equipment | |
| EP3968161A1 (en) | Method and apparatus for generating error reporting content of deep learning framework | |
| CN116049142A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112394920B (en) | Application software development method, platform and electronic equipment | |
| CN115576484A (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN110958243A (en) | Network vulnerability submitting method and device, storage medium and electronic equipment | |
| CN110647331A (en) | Development tool acquisition method and device, storage medium and electronic equipment | |
| CN111176718A (en) | Script online method and device, storage medium and electronic equipment | |
| CN110389862B (en) | Data storage method, device, equipment and storage medium | |
| CN111562958A (en) | Page data display management method and device | |
| CN111104409A (en) | Database processing method and device, storage medium and electronic equipment | |
| CN111046430B (en) | Data processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |