CN117171132A - Data synchronization method, device and medium - Google Patents

Abstract

The present invention provides a data synchronization method, device and medium, which relates to the field of computer technology and is used to solve the problem of migrating data without stopping the machine. The method includes: correlating and configuring the source end information and target end to be data synchronized through a synchronization information table. Information; configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table, parse the binlog of the source-side based on the binlog parsing component to obtain the data changes of the source-side, and store the data changes in the intermediate device; configure the binlog parsing component according to the synchronization information table and the data to be synchronized. The source information for data synchronization is associated with the target information to be synchronized, and the consumer program is started to synchronize the data changes stored in the intermediate device to the target. The invention can formulate an efficient data synchronization strategy from the source end to the target end, obtain, store and synchronize data changes according to the data synchronization strategy through the intermediate device, and can realize non-stop, flexible and controllable data migration from the source end to the target end.

Description

Data synchronization method, device and medium

Technical field

The present invention relates at least to the field of computer technology, and in particular to a data synchronization method, an intermediate device for data synchronization and a computer-readable storage medium.

Background technique

In the traditional data migration process, it is necessary to stop writing data to the source when migrating data to ensure data consistency. This will cause the business to be suspended during the data migration, affecting the service performance and user experience of the business.

Therefore, how to migrate data without downtime and ensure business service performance and user experience is an urgent technical issue in this field that needs to be solved.

Contents of the invention

The technical problem to be solved by the present invention is to provide a data synchronization method, an intermediate device for data synchronization and a computer-readable storage medium to solve the problem of how to migrate data without downtime and ensure business service performance and user experience.

In a first aspect, the present invention provides a data synchronization method, which is applied to an intermediate device connecting a source end and a target end, and includes:

Configure the source information and target information to be synchronized through the synchronization information table;

Configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table, parse the binlog of the source-side based on the binlog parsing component to obtain the data changes of the source-side, and store the data changes in the intermediate device;

According to the target end information to be synchronized that is associated with the source end information to be synchronized in the synchronization information table, start the consumer program to synchronize the data changes stored in the intermediate device to the target end;

Among them, binlog is a binary log file.

Further, the source information and target information to be synchronized are configured through the synchronization information table, specifically including:

Set the source information table, target information table and synchronization information table;

Write multiple source database identifiers in the source information table, and set a first storage location for storing BinlogTopic and a second storage location for storing the name and primary key of the source data table in association with each source database identifier;

writing multiple target end addresses in the target end information table, and setting a third storage location for storing target end partition information in association with each target end address;

Write the source database identifier to be synchronized and the target address to be synchronized to in the synchronization information table;

Among them, BinlogTopic is the unique identifier of the binlog parsing component.

Further, configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table, parse the binlog of the source-side based on the binlog parsing component to obtain the data changes of the source-side, and store the data changes in the intermediate device, including:

Configure the binlog parsing component according to the source database identifier to be synchronized, generate the BinlogTopic of the binlog parsing component, write the BinlogTopic into the first storage location, and set a fourth storage location for storing source data changes based on the BinlogTopic in the intermediate device;

Based on the binlog parsing component, the binlog of the source is parsed to obtain the data changes of the source, the name and primary key of the source data table are obtained according to the data changes, the name and primary key of the source data table are written to the second storage location, and the data changes are written to the second storage location. Four storage locations.

Further, based on the target end information to be synchronized in the synchronization information table associated with the source end information to be synchronized, the consumption program is started to synchronize the data changes stored in the intermediate device to the target end, specifically including:

Obtain the BinlogTopic and the name and primary key of the source data table based on the source database identifier to be synchronized, and set the partition on the target based on the name and primary key of the source data table and the target address to which the source data is to be synchronized. The target partition information is written to the third storage location;

Start the consumer program to obtain the data changes stored in the intermediate device based on the BinlogTopic, obtain the target partition information based on the target address to be synchronized to which the source data is to be synchronized, which is associated with the source information to be synchronized, and obtain the target partition information based on the target address and target partition. The information synchronizes the acquired data changes to the target.

Further, the form of the source end is specifically a MYSQL database, and the form of the target end is specifically at least one of a MYSQL database, a KAFKA cluster, an MQ cluster, and an ES cluster;

Among them, MYSQL is a relational database management system, KAFKA is an open source stream processing platform, MQ is a message queue, and ES is a search server.

Further, the form of the consumption program is specifically one of a virtual machine, a physical machine, and a container.

Further, the method also includes:

In the synchronization information table, the source information to be synchronized is associated with the storage period of the data changes stored in the intermediate device, and the target information to be synchronized is associated with the start time of the consumer program. The start time is at the Within the above retention period, and/or configure the target filter field associated with the target information to be synchronized.

Further, the method also includes:

An alarm is issued in response to monitoring of insufficient performance of the intermediate device, the inability of the binlog parsing component to complete the parsing operation, or the inability of the consumer program to complete the synchronization operation.

In a second aspect, the present invention provides an intermediate device for data synchronization. The intermediate device connects the source end and the target end and includes:

The configuration module is used to configure the source information and target information to be synchronized through the synchronization information table;

The parsing module, connected to the configuration module, is used to configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table. Based on the binlog parsing component, the binlog of the source-side is parsed to obtain the data changes of the source-side, and the data changes are stored in the intermediate device. ;

The synchronization module is connected to the parsing module and is used to start the consumer program to synchronize the data changes stored in the intermediate device to the target based on the target end information to be synchronized in the synchronization information table associated with the source end information to be synchronized;

Among them, binlog is a binary log file.

In a third aspect, the present invention provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is run by a processor, the above-mentioned data synchronization method is implemented.

The present invention provides a data synchronization method, an intermediate device for data synchronization, and a computer-readable storage medium. Through the associated configuration of the synchronization information table, the intermediate device can obtain source information and target information to be synchronized in a timely manner, and then The binlog parsing component is used to obtain the data changes at the source end in real time, and the data changes are stored in the intermediate device. The data changes stored in the intermediate device are then synchronized to the target end through the consumer program. Flexible control of the source end and the target end can be achieved by synchronizing the information table. Associated configuration allows an efficient data synchronization strategy from the source to the target to be developed. The intermediate device obtains, stores and synchronizes data changes according to the data synchronization strategy, enabling non-stop, flexible and controllable data migration from the source to the target. .

Description of drawings

Figure 1 is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention;

Figure 2 is a flow chart of a data synchronization method according to an embodiment of the present invention;

Figure 3 is a schematic structural diagram of an intermediate device for data synchronization according to an embodiment of the present invention;

Figure 4 is a schematic structural diagram of another data synchronization system according to an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings.

It can be understood that the specific embodiments and drawings described here are only used to explain the present invention, but not to limit the present invention.

It can be understood that, without conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

It can be understood that, for convenience of description, only the parts related to the present invention are shown in the drawings of the present invention, and the parts irrelevant to the present invention are not shown in the drawings.

It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units and modules may be integrated into one entity structure.

It can be understood that, provided there is no conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in a sequence different from that marked in the drawings.

It can be understood that the flowcharts and block diagrams of the present invention illustrate the possible implementation architecture, functions and operations of systems, devices, equipment, and methods according to various embodiments of the present invention. Each box in the flow chart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for realizing the specified function. Furthermore, each block or combination of blocks in the block diagrams and flowchart illustrations may be implemented by a hardware-based system that performs the specified functions, or by a combination of hardware and computer instructions.

It can be understood that the units and modules involved in the embodiments of the present invention can be implemented in software or hardware. For example, the units and modules can be located in a processor.

Example 1:

The present invention provides a data synchronization method, which is applied to the intermediate device 2 connecting the source end 1 and the target end 3 as shown in Figure 1, and includes the steps shown in Figure 2:

S21. Configure the source information and target information to be synchronized through the synchronization information table;

S22. Configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table, parse the binlog of the source-side based on the binlog parsing component to obtain the data changes of the source-side, and store the data changes in the intermediate device;

S23. According to the target terminal information to be synchronized in the synchronization information table associated with the source terminal information to be synchronized, start the consumption program to synchronize the data changes stored in the intermediate device to the target terminal;

Among them, binlog is a binary log file.

Specifically, in this embodiment, a data synchronization system as shown in Figure 1 is first established, and an intermediate device is added between the source end and the target end. The structure of the intermediate device is as shown in Figure 3, including steps for implementing The configuration module 21 of S21, the parsing module 22 for implementing step S22, and the synchronization module 23 for implementing step S23 perform associated configuration of the synchronization information table through the configuration module 21, so that the intermediate device can obtain the source end to be synchronized data in a timely manner. information and target end information, and then the parsing module 22 uses the binlog parsing component to obtain the data changes of the source end in real time, and stores the data changes in the intermediate device, and then synchronizes the data changes stored in the intermediate device to the target end through the consumer program, and synchronizes the information The table can realize flexible association configuration between the source and the target, so that an efficient data synchronization strategy from the source to the target can be formulated. The intermediate device can obtain, store and synchronize data changes according to the data synchronization strategy, which can achieve non-stop and flexible Controlled data migration. In addition, the intermediate device in this embodiment stores data changes at the source end and does not store the full amount of data. The data volume is small and the real-time data synchronization speed is fast. The host using the intermediate device stores data changes, has a large capacity, and has high data storage reliability. Through the table The associated information configuration can set multiple targets for synchronization at the same time.

Further, the source information and target information to be synchronized are configured through the synchronization information table, specifically including:

Set the source information table, target information table and synchronization information table;

Write multiple source database identifiers in the source information table, and set a first storage location for storing BinlogTopic and a second storage location for storing the name and primary key of the source data table in association with each source database identifier;

writing multiple target end addresses in the target end information table, and setting a third storage location for storing target end partition information in association with each target end address;

Write the source database identifier to be synchronized and the target address to be synchronized to in the synchronization information table;

Among them, BinlogTopic is the unique identifier of the binlog parsing component.

Specifically, in this embodiment, the data synchronization system can also be expressed as the composition structure shown in Figure 4. The source end is specifically the source end database, and the intermediate device can also be expressed as including a binlog parsing component, a local storage module and a consumption module. Program cluster, the target end is specifically the target end cluster, uses the binlog parsing component of the intermediate device to monitor the binlog of the source end, and obtains the data changes of the source end according to the binlog in real time, stores the data changes in the local storage module, and then consumes the program cluster from the intermediate device Migrate the data changes at the source end to the target end. The local storage module also stores the table association strategy that controls the above data migration process. The table association strategy includes three tables: the source end information table, the target end information table and the synchronization information table. Specifically The association configuration of the source information and target information to be synchronized can be realized through the synchronization information table. The source and target information provided by the business side that need to be synchronized can be obtained through the synchronization information table. The source end and target end need to be synchronized through the table association strategy. Obtain source information and target information from the information table and target information table. The source information table and target information table can be preset for multiple sources and targets connected to the system. First, record the basic source information. and target end information. After receiving the specific data synchronization task from the business side, record the requirements of the data synchronization task in the synchronization information table, record the association between multiple sets of source end and target end, and multiple source end to target end synchronization requirements. .

A more specific example is a scenario where the business is deployed in a single region. If data outage migration occurs, the overall migration process may take several days depending on the amount of data and network delay conditions, and the downtime period of several days may have an impact on the business. It has a great impact. With the rapid development of business and the increase in the number of users, the user experience may be poor due to the high delay of cross-domain access to the business and the inability to access the business in real time, affecting the scalability of the business, and may also occur due to power outages. , network interruption and other force majeure factors lead to the inability to obtain data normally, which in turn leads to data migration failure. It is very necessary to propose a data migration method with excellent performance for this situation. Incremental data synchronization is a reference solution, such as using a massive platform as the source database, Hadoop (a distributed system infrastructure developed by the Apache Foundation) big data platform as the target database, MapReduce (a programming model , used for parallel computing of large-scale data sets (larger than 1TB)) as a computing engine for big data, using HDFS (a distributed file system) distributed file system to store unstructured and semi-structured data, and Hbase (a distributed file system) A distributed, column-oriented open source database) distributed database stores structured data, and combines Java (an object-oriented programming language) to develop an automated incremental data import method to achieve incremental import from the source database to the target database. Although incremental synchronization of data is achieved and long-term business downtime caused by one-time overall data migration is avoided, the real-time performance is poor and cannot meet the real needs of the business. Another data synchronization method based on canal (database synchronization tool), using MYSQL (a relational database management system), Oracle (a relational database management system), etc. as the source database, and KAFKA (an open source stream processing platform) , MYSQL, etc. as the target end, simulating the dump (backup file system) protocol of mysql slave (slave device) and mysql master (master device), parsing binlog (log file in binary format) to achieve the basic requirements for data synchronization, but each The target needs to deploy a set of programs for synchronization. If the business expands to 31 provinces, 31 program clusters need to be deployed for data synchronization. If synchronization is performed to different targets, more clusters will be needed. In this case, cluster resources will be exhausted. It is wasteful and requires continuous querying of the source database, resulting in high database load, which may seriously affect the normal use of the business.

Based on the above scenarios and the problems of possible solutions, this embodiment provides a real-time data synchronization system and synchronization method with non-stop migration and high synchronization efficiency. The method is implemented based on the database binlog, and the parsing and setting of the binlog is completed in the intermediate device. , the intermediate device obtains the incremental data from the source end by parsing the binlog, and then synchronizes the incremental data from the intermediate device to the target end. The intermediate device only needs to save the incremental data, which saves storage space and improves synchronization efficiency. Specifically, the data synchronization system includes: source RDS (Relational Database Service) database (such as MYSQL), database binlog parsing component, target cluster, and consumer program that synchronizes transaction data in the source database to the target in real time. Cluster; the consumer program cluster obtains the data synchronization control strategy through the table association strategy of data synchronization, and controls the time of data synchronization and the target end to which it is synchronized through the data synchronization control policy. The table association strategy controls the newly created or modified source end information table and target end. The source information table, the target information table and the synchronization information table all have an id (Identity, identity) field in the three tables, which uniquely identifies a link. The three tables Relying on ID association, the source information table stores source database information, BinlogTopic, database name, table name, primary key, etc.; the target information table stores different information for different targets. For example, Kafka will store Kafka's cluster information, topic, Partitions, etc., the database will store the data table information corresponding to the source database; the synchronization information table will store the source and corresponding target that need to be synchronized, as well as the consumption time point when data synchronization is started, etc.

Further, configure the binlog parsing component according to the source-side information to be synchronized in the synchronization information table, parse the binlog of the source-side based on the binlog parsing component to obtain the data changes of the source-side, and store the data changes in the intermediate device, including:

Configure the binlog parsing component according to the source database identifier to be synchronized, generate the BinlogTopic of the binlog parsing component, write the BinlogTopic into the first storage location, and set a fourth storage location for storing source data changes based on the BinlogTopic in the intermediate device;

Based on the binlog parsing component, the binlog of the source is parsed to obtain the data changes of the source, the name and primary key of the source data table are obtained according to the data changes, the name and primary key of the source data table are written to the second storage location, and the data changes are written to the second storage location. Four storage locations.

Specifically, in this embodiment, in the data synchronization system based on the database binlog, the form of the source database is specifically a MYSQL database; the database binlog parsing component is used to parse the database binlog information, parse, filter, and standard format the source data. , and store it locally; the consumer program cluster is used to synchronize the consumption of locally stored data in real time, that is, the source data information generated by the database binlog parsing component is processed and synchronized to the target cluster in real time, and the consumer program cluster relies on data synchronization. The table association strategy is used to associate the newly created or modified source information, target information and the required synchronized data table information; the target cluster includes MYSQL database, KAFKA cluster, MQ cluster and other forms.

Further, based on the target end information to be synchronized in the synchronization information table associated with the source end information to be synchronized, the consumption program is started to synchronize the data changes stored in the intermediate device to the target end, specifically including:

Obtain the BinlogTopic and the name and primary key of the source data table based on the source database identifier to be synchronized, and set the partition on the target based on the name and primary key of the source data table and the target address to which the source data is to be synchronized. The target partition information is written to the third storage location;

Start the consumer program to obtain the data changes stored in the intermediate device based on the BinlogTopic, obtain the target partition information based on the target address to be synchronized to which the source data is to be synchronized, which is associated with the source information to be synchronized, and obtain the target partition information based on the target address and target partition. The information synchronizes the acquired data changes to the target.

Specifically, in this embodiment, the entire system operation process is as follows: the intermediate device obtains the source-side MYSQL database identification required for data synchronization provided by the business side, configures the database binlog parsing component according to this MYSQL database identification, and generates a unique database-based The database binlog parses the ID (subsequently called BinlogTopic). Each MySQL operation will be recorded to the binlog. The parsing component monitors real-time changes in the source MYSQL database, parses the binlog corresponding to the database in real time, and converts real-time transaction data (user operations on the source database, Including insert, update, delete, etc.), they are formatted and stored locally (the cluster host corresponding to the parsing component); configure the table association information for data synchronization. First, the user synchronizes the source database information, target synchronization information, and required data. Table information is entered into the storage module of the consumer program cluster, and the MYSQL database required on the business side and the name of the data table to be synchronized are determined based on the table association information. The synchronized data table must have a primary key. The primary key can uniquely identify a piece of data and determine the need for synchronization. The target end. If the target end is a database, you need to ensure that the target end database has built a data table that is consistent with the source end table structure. If the target end is a KAFKA or MQ cluster, you need to ensure that the corresponding topic and partition have been built, and are consistent with the source end table structure. The consumer program cluster network is interconnected; start the consumer program, obtain the source database information in the table association information, obtain the locally stored transaction message of the BinlogTopic corresponding to the source MYSQL database according to the BinlogTopic information in the source database information, and obtain the table association information The data synchronization table information in the data synchronization table filters and processes transaction messages based on the table information in the data synchronization table, obtains the target synchronization information in the table association information, and synchronizes the filtered and processed data to the configured target address in real time. target end. In addition, the transmitted data can specifically adopt the key-value (key-value pair) method, where the key is the field information of the table and the value field value. Through the above process, this embodiment solves the problem of business suspension caused by large data volume and delay during user migration. It can synchronize the changed data of multiple tables concurrently. The overall update of the synchronization link is simple and efficient, and the synchronization efficiency is high. During the synchronization period The business can run normally, and a source database can be configured to synchronize to multiple targets concurrently, which solves the waste of deployment cluster resources and reduces the performance impact on the source database. For example, synchronizing the MYSQL database and KAFKA cluster at the same time can achieve cross-domain synchronization and off-site synchronization. Multi-Active truly realizes synchronized backup of data in one province and multiple provinces.

Further, the form of the source end is specifically a MYSQL database, and the form of the target end is specifically at least one of a MYSQL database, a KAFKA cluster, an MQ cluster, and an ES cluster;

Among them, MYSQL is a relational database management system, KAFKA is an open source stream processing platform, MQ is a message queue, and ES is a search server.

Specifically, in this embodiment, the source database may include a MYSQL database, and the target may include a MYSQL database, a KAFKA cluster, an MQ (Message Queue, message queue) cluster, an ES (Elasticsearch, a search server) cluster, etc., and can simultaneously The source data is synchronized to multiple targets, such as to the RDS database and Kafka cluster at the same time.

Further, the form of the consumption program is specifically one of a virtual machine, a physical machine, and a container.

Specifically, in this embodiment, the consumer program cluster may be composed of virtual machines, physical machines, k8s (Kubernetes, a new distributed architecture solution based on container technology) containers, etc.

Further, the method also includes:

In the synchronization information table, the source information to be synchronized is associated with the storage period of the data changes stored in the intermediate device, and the target information to be synchronized is associated with the start time of the consumer program. The start time is at the Within the above retention period, and/or configure the target filter field associated with the target information to be synchronized.

Specifically, in this embodiment, the table association strategy for data synchronization can also be set as follows: the source information table stores the source RDS database information, program deployment cluster information, etc.; the target information table stores the target address, user name, password, attribute information, etc.; data synchronization table information, which stores source database information, synchronization table information, synchronization mode (such as consumption time point, API (Application Programming Interface) address, etc.), etc. When transmitting data to the target end, you can set field conditions for data filtering, set the data retention period to 7 days, dynamically configure consumption time points, select a certain time for data synchronization, and dynamically configure data tables that need to be synchronized. This enables resumable transmission, that is, the data in the synchronization table of the source database is stored for 7 days. By simply adjusting the consumption time point field in the data synchronization information table of the associated table, a certain consumption time point within 7 days can be dynamically set for data synchronization. If the downstream system temporarily stops the data synchronization task due to business, it can resume the transmission within 7 days, and continue the data synchronization from the last consumption pause time to prevent data loss. The dynamically set synchronization target field condition filtering can realize all provinces. Flexible transfer of data.

Further, the method also includes:

An alarm is issued in response to monitoring of insufficient performance of the intermediate device, the inability of the binlog parsing component to complete the parsing operation, or the inability of the consumer program to complete the synchronization operation.

Specifically, in this embodiment, the following auxiliary steps can also be set to monitor and alarm the database binlog parsing component service, such as monitoring whether the binlogtopic is normal and whether delays occur. Delay means that there may be a large amount of data at the source end. Change, the corresponding BinlogTopic may not be able to parse all operations at one time; for monitoring and alarming of the generated data synchronization task, that is, the alarm of the task of synchronizing data to the target, such as whether the task fails and whether there is a delay, the delay refers to the source If a large amount of data occurs on the end, it may not be synchronized to the target end in time; monitor and alarm the host where the binlog parsing component service is located, and the consumer program cluster host, such as whether the disk of the host where the binlog parsing component is located is full, whether the memory is insufficient, etc.; In addition, the business side can also regularly check whether there is any inconsistency between the MYSQL database and the target cluster. If there is any inconsistency, the cause can be queried for data repair.

Example 2:

As shown in Figures 1 and 3, the present invention provides an intermediate device 2 for data synchronization. The intermediate device 2 connects the source end 1 and the target end 3, and includes:

The configuration module 21 is used to associate and configure the source information and target information to be synchronized through the synchronization information table;

The parsing module 22 is connected to the configuration module 21 and is used to configure the binlog parsing component according to the source end information to be synchronized in the synchronization information table, parse the binlog of the source end based on the binlog parsing component to obtain the data changes of the source end, and store the data changes in intermediate equipment;

The synchronization module 23 is connected to the parsing module 22 and is used to start the consumption program to synchronize the data changes stored in the intermediate device to the target according to the target information to be synchronized in the synchronization information table associated with the source information to be synchronized. end;

Among them, binlog is a binary log file.

Further, the configuration module 21 specifically includes:

The table setting unit is used to set the source information table, target information table and synchronization information table;

The source information table unit is used to write multiple source database identifiers in the source information table, and set the first storage location for storing BinlogTopic, the name of the source data table, and the name of the source data table in association with each source database identifier. The secondary storage location of the primary key;

The target information table unit is used to write multiple target addresses in the target information table, and set a third storage location for storing target partition information in association with each target address;

The synchronization information table unit is used to write the source database identifier to be synchronized and the target address to be synchronized to in the synchronization information table;

Among them, BinlogTopic is the unique identifier of the binlog parsing component.

Further, the parsing module 22 specifically includes:

The BinlogTopic unit is used to configure the binlog parsing component according to the source database identifier to be synchronized, generate the BinlogTopic of the binlog parsing component, write the BinlogTopic to the first storage location, and store the fourth step of the source data changes in the intermediate device based on the BinlogTopic setting. storage location;

The data change information unit is used to parse the binlog of the source end based on the binlog parsing component to obtain the data changes of the source end, obtain the name and primary key of the source end data table based on the data changes, and write the name and primary key of the source end data table to the second storage location. , writing the data changes to the fourth storage location.

Further, the synchronization module 23 specifically includes:

The target partition information unit is used to obtain the BinlogTopic and the name and primary key of the source data table based on the source database identifier to be synchronized, and the name and primary key of the source data table and the target to which the source data is to be synchronized. The address sets the partition on the target side and writes the target side partition information to the third storage location;

The target partition synchronization unit is used to start the consumer program to obtain the data changes stored in the intermediate device based on the BinlogTopic, and obtain the target partition information based on the target address to which the source data is to be synchronized, which is associated with the source information to be synchronized. Synchronize the acquired data changes to the target based on the target address and target partition information.

Further, the form of the source end is specifically a MYSQL database, and the form of the target end is specifically at least one of a MYSQL database, a KAFKA cluster, an MQ cluster, and an ES cluster;

Among them, MYSQL is a relational database management system, KAFKA is an open source stream processing platform, MQ is a message queue, and ES is a search server.

Further, the form of the consumption program is specifically one of a virtual machine, a physical machine, and a container.

Further, the synchronization information table unit is also used for:

In the synchronization information table, the source information to be synchronized is associated with the storage period of the data changes stored in the intermediate device, and the target information to be synchronized is associated with the start time of the consumer program. The start time is at the Within the above retention period, and/or configure the target filter field associated with the target information to be synchronized.

Further, the intermediate device 2 also includes:

The monitoring and alarming unit is used to issue an alarm in response to monitoring of insufficient performance of the intermediate device, the inability of the binlog parsing component to complete the parsing operation, or the inability of the consumer program to complete the synchronization operation.

Example 3:

Embodiment 3 of the present invention provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is run by a processor, the data synchronization method as described in Embodiment 1 is implemented.

The computer-readable storage media includes volatile or nonvolatile, removable storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, computer program modules or other data. or non-removable media. Computer-readable storage media include but are not limited to RAM (Random Access Memory, random access memory), ROM (Read-Only Memory, read-only memory), EEPROM (Electrically Erasable Programmable read only memory, electrically erasable programmable read-only memory) , flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassette, tape, magnetic disk storage or other magnetic storage device, or Any other medium that can be used to store the desired information and that can be accessed by a computer.

In addition, the present invention can also provide a computer device, including a memory and a processor. A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the embodiments. The data synchronization method described in 1.

The memory is connected to the processor, and the memory can be flash memory, read-only memory, or other memories, and the processor can be a central processing unit or a single-chip microcomputer.

Furthermore, the present invention can also provide a data synchronization system as shown in Figure 1 or Figure 4, in which the intermediate device performs the data synchronization method as described in Embodiment 1, and the remaining components and their functions are as described in Embodiment 1. narrate.

Embodiments 1-3 of the present invention provide a data synchronization method, an intermediate device for data synchronization, and a computer-readable storage medium. Through the associated configuration of the synchronization information table, the intermediate device can obtain the source information to be synchronized and the source information to be synchronized in a timely manner. Target end information, and then use the binlog parsing component to obtain the data changes of the source end in real time, and store the data changes in the intermediate device, and then synchronize the data changes stored in the intermediate device to the target end through the consumer program. Through the synchronization information table, the source end can be realized Flexible association configuration with the target end, so that an efficient data synchronization strategy from the source end to the target end can be formulated. The intermediate device obtains, stores and synchronizes data changes according to the data synchronization strategy, achieving non-stop and flexible operation from the source end to the target end. Controlled data migration.

It can be understood that the above embodiments are only exemplary embodiments adopted to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also regarded as the protection scope of the present invention.

Claims (10)

1. The data synchronization method is characterized by being applied to an intermediate device connecting a source end and a target end, and comprising the following steps:

source terminal information and target terminal information to be subjected to data synchronization are configured in an associated mode through a synchronization information table;

configuring a binlog analysis component according to source information to be subjected to data synchronization in a synchronization information table, analyzing binlog of a source based on the binlog analysis component to obtain data change of the source, and storing the data change in an intermediate device;

according to target end information to be subjected to data synchronization, which is associated with source end information to be subjected to data synchronization, in the synchronization information table, starting a consumption program to synchronize data changes stored in the intermediate equipment to the target end;

where binlog is a binary log file.

2. The method according to claim 1, wherein the source side information and the target side information to be subjected to data synchronization are configured in a correlated manner through a synchronization information table, specifically comprising:

setting a source end information table, a target end information table and a synchronous information table;

writing a plurality of source database identifications in a source information table, and setting a first storage position for storing BinlogTopic, a name for storing the source data table and a second storage position for storing a main key in association with each source database identification;

writing a plurality of target end addresses into a target end information table, and setting a third storage position for storing target end partition information in association with each target end address;

writing a source end database identifier to be subjected to data synchronization and a target end address to which source end data are to be synchronized in a related manner in a synchronization information table;

wherein BinlogTopic is the unique identification of the binlog parsing component.

3. The method according to claim 2, wherein the binlog parsing component is configured according to source information to be data synchronized in the synchronization information table, and the binlog of the source is parsed by the binlog parsing component to obtain data changes of the source, and the data changes are stored in the intermediate device, specifically including:

configuring a binlog analysis component according to a source end database identifier to be subjected to data synchronization, generating a binlog topic of the binlog analysis component, writing the binlog topic into a first storage position, and setting a fourth storage position for storing data change of a source end based on the binlog topic in an intermediate device;

and analyzing the binlog of the source terminal based on the binlog analysis component to acquire the data change of the source terminal, acquiring the name and the primary key of the source terminal data table according to the data change, writing the name and the primary key of the source terminal data table into the second storage position, and writing the data change into the fourth storage position.

4. A method according to claim 3, wherein starting the consumption program to synchronize the data change stored in the intermediate device to the target according to the target information to be synchronized associated with the source information to be synchronized in the synchronization information table, specifically comprises:

acquiring the names and the primary keys of the BinlogTopic and the source data table according to the source database identification to be subjected to data synchronization, setting a partition at a target end according to the names and the primary keys of the source data table and the target end address to be subjected to the source data synchronization, and writing target end partition information into a third storage position;

and starting a consumption program to acquire data changes stored in the intermediate device according to the BinlogTopic, acquiring target end partition information according to a target end address to which source end data are to be synchronized, which is associated with the source end information to be subjected to data synchronization, and synchronizing the acquired data changes to the target end according to the target end address and the target end partition information.

5. The method according to any one of claims 1-4, wherein the source terminal is in the form of a MYSQL database, and the target terminal is in the form of at least one of a MYSQL database, a KAFKA cluster, a MQ cluster, and an ES cluster;

wherein MYSQL is a relational database management system, KAFKA is an open source stream processing platform, MQ is a message queue, and ES is a search server.

6. The method according to any of claims 1-4, wherein the consumption program is in the form of one of a virtual machine, a physical machine, a container.

7. The method according to any one of claims 1-4, further comprising:

in the synchronization information table, the source information configuration associated with the data to be synchronized stores the data change in a storage period of the intermediate device, and the target information configuration associated with the data to be synchronized consumes a start time of the program, wherein the start time is within the storage period, and/or the target information configuration associated with the data to be synchronized targets filters.

8. The method according to any one of claims 1-4, further comprising:

and sending an alarm in response to monitoring that the performance of the intermediate device is insufficient, the binlog analysis component cannot complete analysis operation, or the consumption program cannot complete synchronous operation.

9. An intermediate device for data synchronization, wherein the intermediate device connects a source end and a target end, and comprises:

the configuration module is used for associating and configuring source terminal information and target terminal information to be subjected to data synchronization through the synchronization information table;

the analysis module is connected with the configuration module and is used for configuring a binlog analysis component according to source end information to be subjected to data synchronization in the synchronization information table, analyzing binlog of a source end based on the binlog analysis component to acquire data change of the source end, and storing the data change in the intermediate equipment;

the synchronous module is connected with the analysis module and is used for starting a consumption program to synchronize the data change stored in the intermediate equipment to the target end according to the target end information to be subjected to data synchronization, which is associated with the source end information to be subjected to data synchronization, in the synchronous information table;

where binlog is a binary log file.

10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when being executed by a processor, implements the data synchronization method according to any of claims 1-8.