CN117725130A - Data synchronization method and device, storage medium and electronic equipment - Google Patents

Data synchronization method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN117725130A
CN117725130A CN202311748037.8A CN202311748037A CN117725130A CN 117725130 A CN117725130 A CN 117725130A CN 202311748037 A CN202311748037 A CN 202311748037A CN 117725130 A CN117725130 A CN 117725130A
Authority
CN
China
Prior art keywords
target
database
source database
data
tool
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311748037.8A
Other languages
Chinese (zh)
Inventor
徐冬冬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianyi Electronic Commerce Co Ltd
Original Assignee
Tianyi Electronic Commerce Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianyi Electronic Commerce Co Ltd filed Critical Tianyi Electronic Commerce Co Ltd
Priority to CN202311748037.8A priority Critical patent/CN117725130A/en
Publication of CN117725130A publication Critical patent/CN117725130A/en
Pending legal-status Critical Current

Links

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data synchronization method and device, a storage medium and electronic equipment, and relates to the technical field of network security. The method comprises the following steps: determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database; establishing connection between a target tool and a source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of the source database through a target tool, and analyzing the target log to obtain change data of the source database; and establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database. According to the method and the device, the problem that in the related art, the data in the source database are synchronized to the target database in a mode of copying the data at fixed time, so that the real-time performance of the data synchronization is poor is solved.

Description

Data synchronization method and device, storage medium and electronic equipment
Technical Field
The present application relates to the field of network security technologies, and in particular, to a data synchronization method and apparatus, a storage medium, and an electronic device.
Background
Currently, data is one of the most important assets in businesses and organizations. Moreover, many enterprises have distributed systems and multiple database instances where data in the databases needs to be kept consistent to support business operations. However, due to the complexity of a distributed environment, synchronizing data from one database to another becomes challenging.
In the related art, a data synchronization method generally includes periodically exporting and importing data in batch, or copying data using a database copy function. However, the data synchronization method in the related art may result in poor real-time performance of data synchronization.
Aiming at the problem that the real-time property of data synchronization is poor due to the fact that the data in the source database is synchronized to the target database by adopting a mode of copying the data in a timing mode in the related art, no effective solution is proposed at present.
Disclosure of Invention
The main purpose of the application is to provide a data synchronization method and device, a storage medium and an electronic device, so as to solve the problem that in the related art, the real-time performance of data synchronization is poor due to the fact that data in a source database is synchronized into a target database in a mode of copying data at fixed time.
To achieve the above object, according to one aspect of the present application, there is provided a data synchronization method. The method comprises the following steps: determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database; establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of the source database through the target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; and establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database.
Further, the data synchronization method is applied in a data synchronization system, and the data synchronization system at least comprises: the target tool, the secondary database of the source database, and the primary database of the source database, the method further comprising, prior to listening to a target log of the source database by the target tool: acquiring an interaction protocol through the target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of the source database and a master database of the source database; based on the interaction protocol, sending a monitoring request for monitoring the target log to a main database of the source database; and responding to the monitoring request through a main database of the source database, and sending the target log to the target tool.
Further, the analyzing the target log to obtain the change data of the source database includes: acquiring an object code for analyzing the operation information of the source database; executing the target code, and analyzing the target log to obtain information of the change operation of the source database; acquiring original data stored in the source database; and obtaining the change data of the source database based on the original data and the change operation information of the source database.
Further, synchronizing the change data of the source database into the target database includes: performing format conversion on the change data of the source database to obtain converted change data; determining a transmission mode of data transmission between the target tool and the target database; and synchronizing the converted change data to the target database by adopting the transmission mode.
Further, synchronizing the converted change data to the target database by adopting the transmission mode includes: determining a message queue, wherein the message queue is used for transmitting data in the target tool to the target database; and synchronizing the converted change data to the target database by adopting the transmission mode in combination with the message queue.
Further, obtaining the connection information of the source database includes: determining a target host where the source database is located, and acquiring name information of the target host; acquiring a parameter information set of the source database, wherein the parameter information set of the source database at least comprises: the IP address information of the source database, the port number information of the source database, the user name information and the password information for accessing the source database; and acquiring connection information of the source database based on the name information of the target host and the parameter information set of the source database.
Further, after synchronizing the change data of the source database into the target database, the method further comprises: acquiring a data synchronization result, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database to the target database is successful; under the condition that the data synchronization result is that the synchronization of the change data of the source database to the target database fails, the change data of the source database is synchronized again to the target database, and the target times are obtained, wherein the target times are times for synchronizing the change data of the source database to the target database again; judging whether the target times are larger than preset times or not; if the target times are not greater than the preset times, continuing to resynchronize the change data of the source database to the target database; and if the target times are greater than the preset times, sending reminding information to a target object, wherein the reminding information is used for reminding the target object of synchronization failure when the target object synchronizes the change data of the source database into the target database.
To achieve the above object, according to another aspect of the present application, there is provided a data synchronization apparatus. The device comprises: the first processing unit is used for determining a source database and a target database and acquiring connection information of the source database and connection information of the target database; the first establishing unit is used for establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; the second processing unit is used for monitoring a target log of the source database through the target tool, analyzing the target log and obtaining change data of the source database, wherein the target log is used for recording change operation of the source database; and the third processing unit is used for establishing connection between the target tool and the target database according to the connection information of the target database and synchronizing the change data of the source database into the target database.
Further, the data synchronization device is applied in a data synchronization system, and the data synchronization system at least comprises: the target tool, a secondary database of the source database, and a master database of the source database, the apparatus further comprising: the first acquisition unit is used for acquiring an interaction protocol by the target tool before monitoring a target log of the source database by the target tool, wherein the interaction protocol is a protocol used when a slave database of the source database and a master database of the source database perform data interaction; the first sending unit is used for sending a monitoring request for monitoring the target log to a main database of the source database based on the interaction protocol; and the fourth processing unit is used for responding to the monitoring request through the main database of the source database and sending the target log to the target tool.
Further, the second processing unit includes: the first acquisition module is used for acquiring target codes for analyzing the operation information of the source database; the first processing module is used for executing the target code, analyzing the target log and obtaining information of changing operation of the source database; the second acquisition module is used for acquiring the original data stored in the source database; and the first determining module is used for obtaining the change data of the source database based on the original data and the information of the change operation of the source database.
Further, the third processing unit includes: the first conversion module is used for carrying out format conversion on the change data of the source database to obtain converted change data; the second determining module is used for determining a transmission mode of data transmission between the target tool and the target database; and the first synchronization module is used for synchronizing the converted change data to the target database by adopting the transmission mode.
Further, the first synchronization module includes: a first determining submodule for determining a message queue, wherein the message queue is used for transmitting data in the target tool to the target database; and the first synchronization sub-module is used for combining the message queue and synchronizing the converted change data into the target database by adopting the transmission mode.
Further, the first processing unit includes: the second processing module is used for determining a target host where the source database is located and acquiring name information of the target host; the third obtaining module is configured to obtain a parameter information set of the source database, where the parameter information set of the source database at least includes: the IP address information of the source database, the port number information of the source database, the user name information and the password information for accessing the source database; and a fourth acquisition module, configured to acquire connection information of the source database based on name information of the target host and a parameter information set of the source database.
Further, the apparatus further comprises: the second acquisition unit is used for acquiring a data synchronization result after synchronizing the change data of the source database into the target database, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database into the target database is successful or not; a fifth processing unit, configured to, when the data synchronization result is that synchronization of the change data of the source database to the target database fails, re-synchronize the change data of the source database to the target database, and obtain a target number of times, where the target number of times is the number of times that the change data of the source database is re-synchronized to the target database; the first judging unit is used for judging whether the target times are larger than preset times or not; the first synchronization unit is used for continuously and re-synchronizing the change data of the source database to the target database if the target frequency is not greater than the preset frequency; and the second sending unit is used for sending reminding information to a target object if the target frequency is larger than the preset frequency, wherein the reminding information is used for reminding the target object that the synchronization fails when the target object synchronizes the change data of the source database into the target database.
In order to achieve the above object, according to another aspect of the present application, there is provided a computer-readable storage medium storing a program, wherein the program performs the data synchronization method of any one of the above.
To achieve the above object, according to another aspect of the present application, there is provided an electronic device including one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement any one of the data synchronization methods described above.
Through the application, the following steps are adopted: determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database; establishing connection between a target tool and a source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of a source database through a target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; according to the connection information of the target database, connection between the target tool and the target database is established, and the change data of the source database is synchronized into the target database, so that the problem of poor real-time performance of data synchronization caused by synchronizing the data in the source database into the target database in a mode of copying the data at fixed time in the related art is solved. The method comprises the steps of establishing connection between a target tool and a source database according to connection information of the source database, wherein the target tool is a tool for data synchronization, monitoring a target log of the source database by using the target tool, analyzing the target log to obtain change data of the source database, establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database into the target database by using the target tool, so that the change data of the source database can be synchronized into the target database in real time, and further the effect of improving the real-time performance of data synchronization is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, illustrate and explain the application and are not to be construed as limiting the application. In the drawings:
FIG. 1 is a flow chart of a data synchronization method provided in accordance with an embodiment of the present application;
FIG. 2 is a flow chart of an alternative data synchronization method provided in accordance with an embodiment of the present application;
FIG. 3 is a schematic diagram of a data synchronization method provided according to an embodiment of the present application;
FIG. 4 is a flow chart of a failure recovery and retry mechanism provided in accordance with an embodiment of the present application;
FIG. 5 is a schematic diagram of a data synchronization device provided according to an embodiment of the present application;
fig. 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application.
Detailed Description
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, related information (including, but not limited to, user equipment information, user personal information, etc.) and data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by a user or sufficiently authorized by each party. For example, an interface is provided between the system and the relevant user or institution, before acquiring the relevant information, the system needs to send an acquisition request to the user or institution through the interface, and acquire the relevant information after receiving the consent information fed back by the user or institution.
The present invention will be described with reference to preferred implementation steps, and fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present application, as shown in fig. 1, and the method includes the following steps:
step S101, determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database.
For example, the source database may be a MySQL database and the target database may be a MySQL database or other supported database. Moreover, mySQL database is a widely used relational database management system used by many enterprises as its core data storage solution. In addition, the connection information may include a host name, a port number, a user name, a password, and the like. For example, the connection information of the source database may include a hostname, a port number, a user name, a password, etc. of the source database; the connection information of the target database may include a hostname, a port number, a user name, a password, and the like of the target database.
Step S102, connection between a target tool and a source database is established according to the connection information of the source database, wherein the target tool is a tool for data synchronization.
For example, the target tool may be a data synchronization tool, and the data synchronization tool may be an open-source MySQL data capture and replication framework capable of listening to binlog (binary log) of MySQL and capturing database changes. For example, the data synchronization tool may be coupled to the source database based on the hostname, port number, user name, password, etc. of the source database.
Step S103, monitoring a target log of the source database through a target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database.
For example, the target log may be a binlog (binary log) log, and the binlog (binary log) may record the change operation of the database. After the connection between the data synchronization tool and the source database is ensured, the data synchronization tool may be used to monitor the binlog (binary log) of the source database (MySQL database), and then the data synchronization framework may analyze the monitored binlog into specific database operations, such as insertion, update, deletion, etc., and may obtain the change data of the source database according to the analyzed change operation of the source database.
Step S104, establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database.
For example, the synchronization tool may be connected to the target database according to the hostname, port number, user name, password, etc., and the data synchronization framework may then send the parsed data to the target system (target database) for processing and storage.
Through the steps S101 to S104, the connection between the target tool and the source database is established according to the connection information of the source database, the target tool is a tool for performing data synchronization, then the target tool is used for monitoring the target log of the source database, analyzing the target log to obtain the change data of the source database, then the connection between the target tool and the target database is established according to the connection information of the target database, and then the change data of the source database is synchronized to the target database by using the target tool, so that the change data of the source database can be synchronized to the target database in real time, and further the effect of improving the real-time performance of data synchronization is achieved.
Optionally, in the data synchronization method provided in the embodiment of the present application, obtaining connection information of a source database includes: determining a target host where a source database is located, and acquiring name information of the target host; acquiring a parameter information set of a source database, wherein the parameter information set of the source database at least comprises: IP address information of the source database, port number information of the source database, user name information and password information of the access source database; and acquiring the connection information of the source database based on the name information of the target host and the parameter information set of the source database.
For example, the connection information of the source database may include a hostname, a port number, a user name, a password, and the like. For example, when obtaining the connection information of the source database, the host name of the host where the source database is located may be obtained first, and the IP (Internet Protocol ) address of the source database, the port number of the source database, the user name and the password for accessing the source database may be obtained, and then the obtained host name of the host where the source database is located, the IP address of the source database, the port number of the source database, the user name and the password for accessing the source database may be summarized together to obtain the connection information of the source database.
By the scheme, the connection information of the source database can be quickly and accurately acquired.
Optionally, in the data synchronization method provided in the embodiment of the present application, the data synchronization method is applied in a data synchronization system, where the data synchronization system at least includes: the target tool, the secondary database of the source database, and the master database of the source database, the method further comprising, prior to listening to the target log of the source database by the target tool: acquiring an interaction protocol through a target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of a source database and a master database of the source database; based on an interaction protocol, sending a monitoring request for monitoring a target log to a main database of a source database; and responding to the monitoring request through the main database of the source database, and sending a target log to the target tool.
For example, after the data synchronization tool establishes a connection with the source database, the data synchronization tool may simulate the slave (slave) interaction protocol of the MySQL database (source database), masquerade as MySQL slave (slave of MySQL database), and send the forwarding protocol to the MySQL Master (Master of MySQL database) server. Then MySQL Master receives the dump request (snoop request described above) and begins pushing the binary log (target log described above) to slave (i.e., the data synchronization tool).
By the scheme, the binary log for recording the change operation of the source database can be conveniently monitored by using the data synchronization tool.
Optionally, in the data synchronization method provided in the embodiment of the present application, analyzing the target log to obtain the change data of the source database includes: acquiring an object code for analyzing operation information of a source database; executing the target code, and analyzing the target log to obtain the information of the change operation of the source database; acquiring original data stored in a source database; and obtaining the change data of the source database based on the original data and the change operation information of the source database.
For example, a programming language (e.g., java) may be used to write custom data parsing logic. For example, a code statement written in a programming language for parsing data may be obtained first, and then, by executing the code statement, the data in a binary log (the target log described above) may be parsed to obtain a change operation of the source database, such as insertion, update, deletion, and the like. And acquiring the data originally stored in the source database (i.e. the data stored before the change), and then obtaining the change data of the source database according to the change operation of the source database obtained by analysis and the data originally stored in the source database.
In summary, by executing the custom data parsing logic written in the programming language and parsing the binary log of the source database using the logic, the change data of the source database can be quickly and accurately obtained.
Optionally, in the data synchronization method provided in the embodiment of the present application, synchronizing the change data of the source database to the target database includes: performing format conversion on the change data of the source database to obtain converted change data; determining a transmission mode of data transmission between the target tool and the target database; and synchronizing the converted change data to a target database by adopting a transmission mode.
For example, when the change data of the source database is synchronized to the target database, format conversion may be performed on the change data of the source database, and a format identifiable by the target database may be obtained; the transmission mode of transmitting the data in the data synchronization frame to the target database can be determined, for example, an asynchronous transmission mechanism such as a message queue, a Kafka (distributed stream processing platform for constructing a real-time data pipeline and a stream application program) and the like can be selected, and the data with the converted format is transmitted from the data synchronization frame to the target database by using the data transmission mode.
In summary, by using the data transmission manner, data can be conveniently transmitted from the data synchronization frame to the target database.
Optionally, in the data synchronization method provided in the embodiment of the present application, synchronizing the converted change data to the target database by using a transmission manner includes: determining a message queue, wherein the message queue is used for transmitting data in a target tool to a target database; and synchronizing the converted change data to a target database by adopting a transmission mode in combination with the message queue.
For example, if the transmission manner of data in the data synchronization frame to the target database is a Kafka message queue, the format-converted data may be transmitted from the data synchronization frame to the target database by using the Kafka message queue.
Through the scheme, the Kafka message queue can be conveniently adopted for data transmission.
Optionally, in the data synchronization method provided in the embodiment of the present application, after synchronizing the change data of the source database into the target database, the method further includes: acquiring a data synchronization result, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database to the target database is successful; under the condition that the data synchronization result is that the synchronization of the change data of the source database to the target database fails, the change data of the source database is synchronized again to the target database, and the target times are obtained, wherein the target times are times for synchronizing the change data of the source database to the target database again; judging whether the target times are larger than preset times or not; if the target times are not greater than the preset times, continuing to resynchronize the change data of the source database to the target database; if the target times are greater than the preset times, sending reminding information to the target object, wherein the reminding information is used for reminding the target object that the synchronization fails when the target object synchronizes the change data of the source database into the target database.
For example, to address issues such as a failed connection of the synchronization tool to MySQL or abnormal data synchronization, a fault recovery module may be constructed. The module can monitor the running state of the data synchronization tool and the condition of the synchronous data in real time, and can take corresponding measures for processing when faults occur. For example, parameters such as monitoring interval, failure recovery policy (e.g., reconnecting MySQL or resynchronizing data), and number of retries may be set. These measures may include reconnecting MySQL, resynchronizing the data, etc. In this process, a corresponding failure recovery policy and retry mechanism also need to be set to ensure reliability and stability of data synchronization.
For example, the preset number of times may be 5 times. For example, when the synchronization of the change data of the source database to the target database fails, and the number of times of re-synchronizing the change data of the source database to the target database is 3, it may be determined that the number of times of re-synchronizing the change data of the source database to the target database is 3 or less than 5, and then the synchronization of the change data of the source database to the target database may be continued again; if the number of times of re-synchronizing the change data of the source database to the target database is 6, it may be determined that the number of times of re-synchronizing the change data of the source database to the target database is 6 or more than 5, and then manual intervention may be performed, for example, a mode of sending a prompt message to a system maintainer (the target object) may be adopted to prompt the system maintainer that the data synchronization fails, and the subsequent processing may be performed by the system maintainer.
Through the scheme, the reliability and stability of data synchronization can be ensured by constructing the fault recovery module.
For example, the present embodiment provides a data synchronization implementation scheme based on MySQL and a synchronization tool, that is, by combining MySQL and a data synchronization tool, an efficient data synchronization system may be constructed to synchronize changes of a source database to a target database in real time. Moreover, the data synchronization method in the related art has the defects of higher delay, larger resource occupation and the like. The data synchronization technical scheme based on MySQL+data synchronization tools can overcome the problems and provide the following advantages:
1. the real-time performance is high: the data synchronization framework captures the change operation of the database almost in real time by monitoring the binlog of MySQL, thereby realizing the real-time performance of data synchronization.
2. Accurate capture change: the data synchronization tool is able to parse binlog, converting each operation (e.g., insert, update, delete) into an easily understood data format for further processing and synchronization.
3. Flexible configuration and data filtering: the data synchronization tool provides flexible configuration options, and can analyze, filter and convert data according to actual requirements so as to adapt to different service scenes.
4. Asynchronous transmission and reliability: by using an asynchronous transmission mechanism such as a message queue or Kafka, the change data of the source database can be reliably transmitted to the target system, and retry and recovery can be performed even if a transmission failure occurs.
Therefore, the data synchronization technical scheme based on the MySQL+data synchronization tool has wide significance in the application fields of a distributed system and data consistency. The method provides an efficient and reliable way for enterprises to maintain data consistency among a plurality of databases and supports real-time business decisions and analysis.
For example, by the method provided by the embodiment of the application, high-efficiency and reliable data synchronization can be realized, and the requirement of data consistency among a plurality of databases in a distributed environment can be met; while real-time data updates and operations may be supported.
For example, the implementation scheme of the data synchronization technology based on mysql+data synchronization tool can realize efficient and reliable data synchronization, and ensure data consistency among multiple databases, which is specifically as follows:
1. real-time data synchronization: the data synchronization framework is used for capturing the change operation of the MySQL database in real time and synchronizing the changes to the target system so as to realize the data real-time synchronization between the source database and the target database.
2. Data consistency: ensuring that data between the various databases remains consistent in a distributed environment, data can be properly synchronized and updated even with high concurrency and large data volumes.
3. Flexibility and configurability: flexible configuration options are provided, so that a user can analyze, filter and convert data according to specific service requirements so as to adapt to different data synchronization scenes.
4. Reliability and failure recovery: by adopting the technical means of an asynchronous transmission mechanism, a message queue and the like, the reliability of data synchronization is ensured, and retry and recovery can be performed when faults occur in the transmission process, so that the risk of data loss or inconsistency is reduced.
5. High performance and low latency: by optimizing algorithms and mechanisms for data synchronization and utilizing efficient capture and parsing capabilities of the data synchronization framework, high performance data synchronization is achieved while maintaining low synchronization latency.
6. Compatibility and ease of use: the scheme has good compatibility and supports integration with the existing MySQL database and the target system. Meanwhile, a simple and easy-to-use management interface and tool are provided, so that a user can conveniently configure and monitor the data synchronization process.
Moreover, the specific technical method provided in this embodiment is as follows:
(1) Determining a source database and a target database: a source database (MySQL) and a target database (which may be MySQL or other supported databases) are determined and their connection information is obtained, including hostname, port number, user name, password, etc.
(2) Installation and configuration data synchronization framework
1) The data synchronization framework is downloaded and installed according to guidelines provided by the official documents.
2) The data synchronization framework instance is configured to specify connection information of the source database to be monitored and the binlog log location to be captured.
(3) Starting a data synchronization service: starting the data synchronization server, ensuring that the data synchronization tool establishes connection with the source database, and successfully monitoring the binlog.
(4) Writing data parsing and conversion logic
1) The data synchronization framework will parse the captured binlog into specific database operations, such as insert, update, delete, etc. Logic is written to parse and process these operations according to business requirements.
2) Custom data parsing and conversion logic may be written using a programming language such as Java.
(5) Configuring target system reception
1) The data transmission mode is determined, and a message queue, an asynchronous transmission mechanism such as Kafka and the like can be selected.
2) And carrying out relevant configuration according to the requirements of the target system, and ensuring that the target system can receive and process the data transmitted from the data synchronization tool.
(6) Data synchronization and transmission
1) The data synchronization framework sends the parsed data to the target system for processing and storage.
2) The corresponding message queue or asynchronous transmission mechanism is used for ensuring that the data can be reliably transmitted to the target system, and the system is provided with a fault recovery and retry mechanism.
(7) Monitoring and management
1) And the configuration monitoring and management tool is used for monitoring information such as the data synchronization state, the delay condition and the error log in real time.
2) And designing a proper alarm mechanism so as to discover and solve abnormal conditions in the data synchronization process in time.
(8) Testing and optimization
1) Testing is performed to ensure that the data can be synchronized correctly and in real time between the source database and the target database.
2) Performance optimization can be performed according to actual conditions, such as adjusting a data transmission mode, increasing concurrent processing capacity and the like, so as to improve efficiency and stability of data synchronization.
For example, fig. 2 is a flowchart of an optional data synchronization method provided according to an embodiment of the present application, and as shown in fig. 2, the flow of the optional data synchronization method may be specifically divided into a synchronization tool server and a synchronization tool client, where the synchronization tool server may monitor a binlog of a MySQL database, and analyze the binlog of the MySQL database to obtain change data of the MySQL database, and then synchronize the monitored change data of the MySQL database to the synchronization tool client through the synchronization tool server, and then synchronize the change data of the MySQL database to a target database through the synchronization tool client, and the target database may be a rediss database (an open-source memory database, which may be used as a database, a cache and a message queue), an MQ database (Message Queuing Database, a message queue database, a database system for storing, managing and delivering messages), a postql database (an open-source relational database management system), and so on.
For example, fig. 3 is a schematic diagram of a data synchronization method provided according to an embodiment of the present application, and as shown in fig. 3, the data synchronization method provided by the embodiment of the present application may include the following steps:
1. the data synchronization tool simulates the interactive protocol of the slave of MySQL, masquerades as MySQL slave, and sends the forwarding protocol to the MySQL Master server.
2. MySQL Master receives the dump request and begins pushing the binary log to slave (i.e., the data synchronization tool).
3. The data synchronization tool parses the binary log object into its own data type (original byte stream).
In addition, the present embodiment has the following advantages and effects:
(1) Real-time performance: the MySQL+data synchronization tool can realize real-time data synchronization, and can timely capture the change of the source database by monitoring the binlog of the database, synchronize the change data to the target database and ensure the latest of the data.
(2) High efficiency: the MySQL+data synchronization tool performs synchronization by using an incremental data extraction mode, only transmits changed data, reduces redundant operation of data transmission and improves efficiency of data synchronization compared with a traditional full replication mode or a polling query mode.
(3) And (3) fine control: the MySQL+data synchronization tool can selectively synchronize the appointed database table, field or event type, flexibly configure according to service requirements, and realize refined data synchronization control.
(4) Scalability: the MySQL+data synchronization tool supports various data consumption modes, can send the synchronized data to different target systems, such as a message queue, a search engine and the like, and is convenient for constructing complex data processing flows and application scenes.
(5) Data consistency and integrity: and the data synchronization is performed by analyzing the binlog of the database, so that the consistency and the integrity of the data are ensured. Because the change log of the database is directly read, the data synchronization error caused by network faults or other factors is avoided.
(6) Incremental backup and restore: the MySQL + data synchronization tool may record the change data as part of the incremental backup, facilitating a quick restore to the data state at the specified point in time.
For example, fig. 4 is a flowchart of a failure recovery and retry mechanism provided according to an embodiment of the present application, and as shown in fig. 4, the failure recovery and retry mechanism provided according to an embodiment of the present application includes the following steps:
(1) Installation and configuration of data synchronization tools: first, the source code of the data synchronization tool is downloaded, compiled and packaged. Then, the basic parameters (such as thread number, buffer size, etc.) of the data synchronization tool are set according to the packaged file for installation and configuration.
And the step of installing and configuring the data synchronization tool comprises:
1) The source code of the data synchronization tool is downloaded.
2) The downloaded source code is decompressed.
3) Enter the decompressed directory and execute the commands in the command line to compile and package the data synchronization framework.
4) The basic parameters of the data synchronization tool are configured. These parameters may be set in a configuration file (e.g., my. Cnf).
(2) Connection and configuration of data synchronization tool with MySQL: after the data synchronization tool is installed and configured, a connection relationship between the data synchronization tool and MySQL needs to be established. In this process, connection information of MySQL database, such as IP address, port number, user name, password, etc., needs to be provided. Meanwhile, rules and policies for reading MySQL data by the data synchronization tool are required to be set.
And the step of connecting and configuring the data synchronization tool with MySQL comprises:
1) In the configuration file (such as my. Cnf) of the data synchronization tool, relevant parameters of the data synchronization tool connection MySQL, such as the IP address, port number, user name, password, etc., are set.
2) In the configuration file (e.g., my. Cnf) of the data synchronization tool, rules and policies are set for the data synchronization tool to read MySQL data. For example, it may be specified which tables need to be synchronized, as well as the timestamp of the synchronization, etc.
(3) Construction and configuration of a fault recovery module: in order to deal with the problems of failure in connection of the data synchronization tool with MySQL or abnormal data synchronization, a fault recovery module needs to be constructed. The module can monitor the running state of the data synchronization tool and the condition of the synchronous data in real time, and can take corresponding measures for processing when faults occur. Parameters such as monitoring interval, fault recovery policy (e.g. reconnecting MySQL or resynchronizing data) and number of retries are set. These measures may include reconnecting MySQL, resynchronizing the data, etc. In this process, a corresponding failure recovery policy and retry mechanism also need to be set to ensure reliability and stability of data synchronization.
(4) System testing and verification: after the above-described installation, configuration, and construction are completed, the system needs to be tested and verified. Some test data can be selected for data synchronization, and the accuracy and stability of the data synchronization can be checked. Meanwhile, the function and effect of the fault recovery module need to be tested, and the success rate and efficiency of fault recovery are verified.
And the steps of system testing and verification include:
1) And selecting some test data to perform data synchronization test.
2) The accuracy and stability of the data synchronization is checked.
3) And when faults occur, testing the functions and effects of the fault recovery module.
4) And verifying the success rate and the efficiency of fault recovery.
In summary, according to the data synchronization method provided by the embodiment of the application, the source database and the target database are determined, and the connection information of the source database and the connection information of the target database are obtained; establishing connection between a target tool and a source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of a source database through a target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; according to the connection information of the target database, connection between the target tool and the target database is established, and the change data of the source database is synchronized into the target database, so that the problem of poor real-time performance of data synchronization caused by synchronizing the data in the source database into the target database in a mode of copying the data at fixed time in the related art is solved. The method comprises the steps of establishing connection between a target tool and a source database according to connection information of the source database, wherein the target tool is a tool for data synchronization, monitoring a target log of the source database by using the target tool, analyzing the target log to obtain change data of the source database, establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database into the target database by using the target tool, so that the change data of the source database can be synchronized into the target database in real time, and further the effect of improving the real-time performance of data synchronization is achieved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment of the application also provides a data synchronization device, and it should be noted that the data synchronization device of the embodiment of the application can be used for executing the data synchronization method provided by the embodiment of the application. The following describes a data synchronization device provided in an embodiment of the present application.
Fig. 5 is a schematic diagram of a data synchronization device according to an embodiment of the present application. As shown in fig. 5, the apparatus includes: a first processing unit 501, a first setup unit 502, a second processing unit 503, and a third processing unit 504.
Specifically, a first processing unit 501 is configured to determine a source database and a target database, and obtain connection information of the source database and connection information of the target database;
a first establishing unit 502, configured to establish a connection between a target tool and a source database according to connection information of the source database, where the target tool is a tool for performing data synchronization;
A second processing unit 503, configured to monitor, by using a target tool, a target log of the source database, and analyze the target log to obtain change data of the source database, where the target log is used to record a change operation of the source database;
the third processing unit 504 is configured to establish a connection between the target tool and the target database according to the connection information of the target database, and synchronize the change data of the source database to the target database.
In summary, in the data synchronization device provided in the embodiments of the present application, a source database and a target database are determined by a first processing unit 501, and connection information of the source database and connection information of the target database are obtained; the first establishing unit 502 establishes connection between a target tool and a source database according to connection information of the source database, wherein the target tool is a tool for performing data synchronization; the second processing unit 503 monitors a target log of the source database through a target tool, and analyzes the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; the third processing unit 504 establishes connection between the target tool and the target database according to the connection information of the target database, and synchronizes the changed data of the source database to the target database, thereby solving the problem of poor real-time performance of data synchronization caused by synchronizing the data in the source database to the target database by adopting a mode of copying the data at fixed time in the related art. The method comprises the steps of establishing connection between a target tool and a source database according to connection information of the source database, wherein the target tool is a tool for data synchronization, monitoring a target log of the source database by using the target tool, analyzing the target log to obtain change data of the source database, establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database into the target database by using the target tool, so that the change data of the source database can be synchronized into the target database in real time, and further the effect of improving the real-time performance of data synchronization is achieved.
Optionally, in the data synchronization device provided in the embodiment of the present application, the data synchronization device is applied in a data synchronization system, where the data synchronization system at least includes: a target tool, a secondary database of a source database, and a master database of the source database, the apparatus further comprising: the first acquisition unit is used for acquiring an interaction protocol by the target tool before monitoring a target log of the source database by the target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of the source database and a master database of the source database; the first sending unit is used for sending a monitoring request of a monitoring target log to a main database of the source database based on the interaction protocol; and the fourth processing unit is used for responding to the monitoring request through the main database of the source database and sending a target log to the target tool.
Optionally, in the data synchronization device provided in the embodiment of the present application, the second processing unit includes: the first acquisition module is used for acquiring target codes for analyzing the operation information of the source database; the first processing module is used for executing the target code, analyzing and processing the target log to obtain the information of the change operation of the source database; the second acquisition module is used for acquiring the original data stored in the source database; the first determining module is used for obtaining the change data of the source database based on the original data and the change operation information of the source database.
Optionally, in the data synchronization device provided in the embodiment of the present application, the third processing unit includes: the first conversion module is used for carrying out format conversion on the change data of the source database to obtain converted change data; the second determining module is used for determining a transmission mode of data transmission between the target tool and the target database; and the first synchronization module is used for synchronizing the converted change data to the target database by adopting a transmission mode.
Optionally, in the data synchronization device provided in the embodiment of the present application, the first synchronization module includes: a first determining submodule for determining a message queue, wherein the message queue is used for transmitting data in a target tool to a target database; and the first synchronization sub-module is used for synchronizing the converted change data to the target database by adopting a transmission mode in combination with the message queue.
Optionally, in the data synchronization device provided in the embodiment of the present application, the first processing unit includes: the second processing module is used for determining a target host where the source database is located and acquiring name information of the target host; the third obtaining module is configured to obtain a parameter information set of the source database, where the parameter information set of the source database at least includes: IP address information of the source database, port number information of the source database, user name information and password information of the access source database; and the fourth acquisition module is used for acquiring the connection information of the source database based on the name information of the target host and the parameter information set of the source database.
Optionally, in the data synchronization device provided in the embodiment of the present application, the device further includes: the second acquisition unit is used for acquiring a data synchronization result after synchronizing the change data of the source database into the target database, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database into the target database is successful or not; a fifth processing unit, configured to, when the data synchronization result is that synchronization of the change data of the source database to the target database fails, re-synchronize the change data of the source database to the target database, and obtain a target number of times, where the target number of times is the number of times that the change data of the source database is re-synchronized to the target database; the first judging unit is used for judging whether the target times are larger than preset times or not; the first synchronization unit is used for continuously and re-synchronizing the change data of the source database to the target database if the target frequency is not greater than the preset frequency; and the second sending unit is used for sending reminding information to the target object if the target frequency is greater than the preset frequency, wherein the reminding information is used for reminding the target object that the synchronization fails when the target object synchronizes the change data of the source database into the target database.
The data synchronizing device includes a processor and a memory, where the first processing unit 501, the first establishing unit 502, the second processing unit 503, the third processing unit 504, and the like are stored as program units, and the processor executes the program units stored in the memory to implement corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the real-time performance of data synchronization is improved by adjusting the kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program which, when executed by a processor, implements the data synchronization method.
The embodiment of the invention provides a processor which is used for running a program, wherein the data synchronization method is executed when the program runs.
As shown in fig. 6, an embodiment of the present invention provides an electronic device, where the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, the following steps are implemented: determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database; establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of the source database through the target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; and establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database.
The processor also realizes the following steps when executing the program: the data synchronization method is applied to a data synchronization system, and the data synchronization system at least comprises: the target tool, the secondary database of the source database, and the primary database of the source database, the method further comprising, prior to listening to a target log of the source database by the target tool: acquiring an interaction protocol through the target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of the source database and a master database of the source database; based on the interaction protocol, sending a monitoring request for monitoring the target log to a main database of the source database; and responding to the monitoring request through a main database of the source database, and sending the target log to the target tool.
The processor also realizes the following steps when executing the program: analyzing the target log to obtain change data of the source database, wherein the change data comprises the following steps: acquiring an object code for analyzing the operation information of the source database; executing the target code, and analyzing the target log to obtain information of the change operation of the source database; acquiring original data stored in the source database; and obtaining the change data of the source database based on the original data and the change operation information of the source database.
The processor also realizes the following steps when executing the program: synchronizing the change data of the source database into the target database includes: performing format conversion on the change data of the source database to obtain converted change data; determining a transmission mode of data transmission between the target tool and the target database; and synchronizing the converted change data to the target database by adopting the transmission mode.
The processor also realizes the following steps when executing the program: synchronizing the converted change data to the target database by adopting the transmission mode comprises the following steps: determining a message queue, wherein the message queue is used for transmitting data in the target tool to the target database; and synchronizing the converted change data to the target database by adopting the transmission mode in combination with the message queue.
The processor also realizes the following steps when executing the program: the obtaining the connection information of the source database comprises the following steps: determining a target host where the source database is located, and acquiring name information of the target host; acquiring a parameter information set of the source database, wherein the parameter information set of the source database at least comprises: the IP address information of the source database, the port number information of the source database, the user name information and the password information for accessing the source database; and acquiring connection information of the source database based on the name information of the target host and the parameter information set of the source database.
The processor also realizes the following steps when executing the program: after synchronizing the change data of the source database into the target database, the method further comprises: acquiring a data synchronization result, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database to the target database is successful; under the condition that the data synchronization result is that the synchronization of the change data of the source database to the target database fails, the change data of the source database is synchronized again to the target database, and the target times are obtained, wherein the target times are times for synchronizing the change data of the source database to the target database again; judging whether the target times are larger than preset times or not; if the target times are not greater than the preset times, continuing to resynchronize the change data of the source database to the target database; and if the target times are greater than the preset times, sending reminding information to a target object, wherein the reminding information is used for reminding the target object of synchronization failure when the target object synchronizes the change data of the source database into the target database.
The device herein may be a server, PC, PAD, cell phone, etc.
The present application also provides a computer program product adapted to perform, when executed on a data processing device, a program initialized with the method steps of: determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database; establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization; monitoring a target log of the source database through the target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database; and establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: the data synchronization method is applied to a data synchronization system, and the data synchronization system at least comprises: the target tool, the secondary database of the source database, and the primary database of the source database, the method further comprising, prior to listening to a target log of the source database by the target tool: acquiring an interaction protocol through the target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of the source database and a master database of the source database; based on the interaction protocol, sending a monitoring request for monitoring the target log to a main database of the source database; and responding to the monitoring request through a main database of the source database, and sending the target log to the target tool.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: analyzing the target log to obtain change data of the source database, wherein the change data comprises the following steps: acquiring an object code for analyzing the operation information of the source database; executing the target code, and analyzing the target log to obtain information of the change operation of the source database; acquiring original data stored in the source database; and obtaining the change data of the source database based on the original data and the change operation information of the source database.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: synchronizing the change data of the source database into the target database includes: performing format conversion on the change data of the source database to obtain converted change data; determining a transmission mode of data transmission between the target tool and the target database; and synchronizing the converted change data to the target database by adopting the transmission mode.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: synchronizing the converted change data to the target database by adopting the transmission mode comprises the following steps: determining a message queue, wherein the message queue is used for transmitting data in the target tool to the target database; and synchronizing the converted change data to the target database by adopting the transmission mode in combination with the message queue.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: the obtaining the connection information of the source database comprises the following steps: determining a target host where the source database is located, and acquiring name information of the target host; acquiring a parameter information set of the source database, wherein the parameter information set of the source database at least comprises: the IP address information of the source database, the port number information of the source database, the user name information and the password information for accessing the source database; and acquiring connection information of the source database based on the name information of the target host and the parameter information set of the source database.
When executed on a data processing device, is further adapted to carry out a program initialized with the method steps of: after synchronizing the change data of the source database into the target database, the method further comprises: acquiring a data synchronization result, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database to the target database is successful; under the condition that the data synchronization result is that the synchronization of the change data of the source database to the target database fails, the change data of the source database is synchronized again to the target database, and the target times are obtained, wherein the target times are times for synchronizing the change data of the source database to the target database again; judging whether the target times are larger than preset times or not; if the target times are not greater than the preset times, continuing to resynchronize the change data of the source database to the target database; and if the target times are greater than the preset times, sending reminding information to a target object, wherein the reminding information is used for reminding the target object of synchronization failure when the target object synchronizes the change data of the source database into the target database.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (10)

1. A method of data synchronization, comprising:
determining a source database and a target database, and acquiring connection information of the source database and connection information of the target database;
establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization;
Monitoring a target log of the source database through the target tool, and analyzing the target log to obtain change data of the source database, wherein the target log is used for recording change operation of the source database;
and establishing connection between the target tool and the target database according to the connection information of the target database, and synchronizing the change data of the source database to the target database.
2. The method according to claim 1, wherein the data synchronization method is applied in a data synchronization system, and the data synchronization system at least comprises: the target tool, the secondary database of the source database, and the primary database of the source database, the method further comprising, prior to listening to a target log of the source database by the target tool:
acquiring an interaction protocol through the target tool, wherein the interaction protocol is a protocol used when data interaction is carried out between a slave database of the source database and a master database of the source database;
based on the interaction protocol, sending a monitoring request for monitoring the target log to a main database of the source database;
And responding to the monitoring request through a main database of the source database, and sending the target log to the target tool.
3. The method of claim 1, wherein parsing the target log to obtain change data for the source database comprises:
acquiring an object code for analyzing the operation information of the source database;
executing the target code, and analyzing the target log to obtain information of the change operation of the source database;
acquiring original data stored in the source database;
and obtaining the change data of the source database based on the original data and the change operation information of the source database.
4. The method of claim 1, wherein synchronizing change data of the source database into the target database comprises:
performing format conversion on the change data of the source database to obtain converted change data;
determining a transmission mode of data transmission between the target tool and the target database;
and synchronizing the converted change data to the target database by adopting the transmission mode.
5. The method of claim 4, wherein synchronizing the converted change data into the target database using the transmission scheme comprises:
determining a message queue, wherein the message queue is used for transmitting data in the target tool to the target database;
and synchronizing the converted change data to the target database by adopting the transmission mode in combination with the message queue.
6. The method of claim 1, wherein obtaining connection information for the source database comprises:
determining a target host where the source database is located, and acquiring name information of the target host;
acquiring a parameter information set of the source database, wherein the parameter information set of the source database at least comprises: the IP address information of the source database, the port number information of the source database, the user name information and the password information for accessing the source database;
and acquiring connection information of the source database based on the name information of the target host and the parameter information set of the source database.
7. The method of claim 1, wherein after synchronizing the change data of the source database into the target database, the method further comprises:
Acquiring a data synchronization result, wherein the data synchronization result is used for representing whether the synchronization of the change data of the source database to the target database is successful;
under the condition that the data synchronization result is that the synchronization of the change data of the source database to the target database fails, the change data of the source database is synchronized again to the target database, and the target times are obtained, wherein the target times are times for synchronizing the change data of the source database to the target database again;
judging whether the target times are larger than preset times or not;
if the target times are not greater than the preset times, continuing to resynchronize the change data of the source database to the target database;
and if the target times are greater than the preset times, sending reminding information to a target object, wherein the reminding information is used for reminding the target object of synchronization failure when the target object synchronizes the change data of the source database into the target database.
8. A data synchronization device, comprising:
the first processing unit is used for determining a source database and a target database and acquiring connection information of the source database and connection information of the target database;
The first establishing unit is used for establishing connection between a target tool and the source database according to the connection information of the source database, wherein the target tool is a tool for data synchronization;
the second processing unit is used for monitoring a target log of the source database through the target tool, analyzing the target log and obtaining change data of the source database, wherein the target log is used for recording change operation of the source database;
and the third processing unit is used for establishing connection between the target tool and the target database according to the connection information of the target database and synchronizing the change data of the source database into the target database.
9. A computer-readable storage medium storing a program, wherein the program performs the data synchronization method of any one of claims 1 to 7.
10. An electronic device comprising one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data synchronization method of any of claims 1-7.
CN202311748037.8A 2023-12-18 2023-12-18 Data synchronization method and device, storage medium and electronic equipment Pending CN117725130A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311748037.8A CN117725130A (en) 2023-12-18 2023-12-18 Data synchronization method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311748037.8A CN117725130A (en) 2023-12-18 2023-12-18 Data synchronization method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN117725130A true CN117725130A (en) 2024-03-19

Family

ID=90208553

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311748037.8A Pending CN117725130A (en) 2023-12-18 2023-12-18 Data synchronization method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN117725130A (en)

Similar Documents

Publication Publication Date Title
CN109460349B (en) Test case generation method and device based on log
CN104951474B (en) Method and device for acquiring MySQL binlog incremental log
CN110347746B (en) Heterogeneous database synchronous data consistency checking method and device
US9286368B2 (en) Linking framework for information technology management
RU2767149C2 (en) Method and configuration for automated testing system
US20150263909A1 (en) System and method for monitoring a large number of information processing devices in a communication network
EP2429134A1 (en) Method and apparatus for checking and synchronizing data block in distributed file system
CN110825420A (en) Configuration parameter updating method, device, equipment and storage medium for distributed cluster
WO2014090002A1 (en) Data synchronization method, device and system
CN112817791A (en) Mobile terminal monitoring method for mining state of working face cluster
CN110895488B (en) Task scheduling method and device
CN110569142A (en) ORACLE data increment synchronization system and method
CN111338893A (en) Process log processing method and device, computer equipment and storage medium
CN113704790A (en) Abnormal log information summarizing method and computer equipment
CN111339118A (en) Kubernetes-based resource change history recording method and device
CN113010607A (en) Method, device, computer system and storage medium for data synchronization between systems
CN117725130A (en) Data synchronization method and device, storage medium and electronic equipment
CN111241044B (en) Method, device, equipment and readable storage medium for constructing heterogeneous database
CN113407629A (en) Data synchronization method and device, electronic equipment and storage medium
CN112199329A (en) Double-machine operation and maintenance data filing and cleaning method and equipment based on fortress machine
CN112416655A (en) Storage disaster recovery system based on enterprise service portal and data copying method
CN112217881B (en) File synchronization method and equipment based on operation and maintenance audit system
CN111522688A (en) Data backup method and device for distributed system
CN117349384B (en) Database synchronization method, system and equipment
WO2024098746A1 (en) Test method, apparatus, and related device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination