CN117131128A

CN117131128A - Data synchronization method, device, computer equipment and storage medium

Info

Publication number: CN117131128A
Application number: CN202310422736.7A
Authority: CN
Inventors: 吴嘉民; 林恭锐; 肖忠杰; 张丹丹
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-04-19
Filing date: 2023-04-19
Publication date: 2023-11-28

Abstract

The application relates to a data synchronization method, a data synchronization device, computer equipment, a storage medium and a computer program product, and relates to the technical field of databases. Can be used in the field of financial science and technology or other related fields. The method comprises the following steps: analyzing a data operation log of a source database to be expanded, and pushing synchronous operation messages corresponding to at least one data table change operation to message queues corresponding to target message subjects; generating consumer program configuration information according to data table identifiers corresponding to all the data tables in the source database; generating consumer programs subscribing to different message topics according to the consumer program configuration information; executing each consumer program; the consumer program is used for pulling the target synchronous operation message from the message queue corresponding to the subscribed message theme, and executing the data table changing operation matched with the target synchronous operation message in the sub-database of the target database according to the corresponding routing information. The database capacity expansion efficiency can be improved by adopting the method.

Description

Data synchronization method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of database technologies, and in particular, to a database data synchronization method, apparatus, computer device, storage medium, and computer program product.

Background

With the continuous development of internet technology, the scale and the concurrent access number of enterprise-level databases are rapidly increasing, and distributed databases are rapidly and widely applied. A distributed database is a plurality of interconnected databases, typically located at multiple server-sides, that communicate with each other to achieve a common goal.

In order to meet business requirements for large-scale data, it is necessary to expand the capacity of the distributed database. In the existing distributed system database capacity expansion scheme, system shutdown is needed, and after the whole data is imported into a new capacity-expanded database, a background program is started to directly screen and clean the data of the new and old databases. Database expansion can take a long time when the data magnitude is large.

Therefore, there is a problem in the related art that the database expansion efficiency is low in the database expansion.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data synchronization method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve data synchronization efficiency.

In a first aspect, the present application provides a data synchronization method. The method comprises the following steps:

analyzing a data operation log of a source database to be expanded, and pushing synchronous operation information corresponding to at least one data table change operation to an information queue corresponding to a target information subject aiming at least one data table change operation of the source database; the target message topic is a message topic corresponding to a data table operated by the data table changing operation;

Generating consumer program configuration information according to data table identifiers corresponding to all the data tables in the source database; the consumer program configuration information comprises mapping relations between data tables in the source database and message consumption configuration information; the message consumption configuration information comprises a message theme and routing information;

generating consumer programs subscribing to different message topics according to the consumer program configuration information;

executing each of the consumer programs; the consumer program is used for pulling a target synchronous operation message from a message queue corresponding to the subscribed message subject, and executing a data table changing operation matched with the target synchronous operation message in a sub-database of a target database according to the corresponding routing information; the target database is a capacity expansion database corresponding to the source database.

In one embodiment, the generating the configuration information of the consumer program according to the data table identifier corresponding to each data table in the source database includes:

acquiring a data table identifier corresponding to each data table in the source database;

establishing a first mapping relation between each data table identifier and each message theme, and establishing a second mapping relation between each data table identifier and each routing information;

And generating the consumer program configuration information according to the first mapping relation and the second mapping relation.

In one embodiment, the establishing a first mapping relationship between each of the data table identifiers and each of the message topics includes:

obtaining a message theme format template;

supplementing the data table identifiers corresponding to the data tables to the message topic format template to obtain the message topics corresponding to the data tables;

and obtaining the first mapping relation according to the message subjects corresponding to the data tables.

In one embodiment, the performing, in a database of a target database, a data table changing operation matched with the target synchronization operation message according to the corresponding routing information includes:

determining a sub-library of an operation to be executed in the target database according to the corresponding routing information, and taking the sub-library as a target sub-library corresponding to the target synchronous operation message;

and executing the data table changing operation matched with the target synchronous operation message in the target sub-library corresponding to the target synchronous operation message.

In one embodiment, the routing information includes a routing field and a routing algorithm; the determining the sub-database of the operation to be executed in the target database according to the corresponding routing information comprises the following steps:

Calculating corresponding routing fields by adopting a corresponding routing algorithm, and determining a database identification;

and taking the sub-database corresponding to the sub-database identifier in the target database as the sub-database of the operation to be executed.

In one embodiment, the parsing the data operation log of the source database to be expanded, for at least one data table changing operation of the source database, pushing a synchronization operation message corresponding to the at least one data table changing operation to a message queue corresponding to a target message subject, includes:

analyzing the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database;

and packaging the change operation record into a message in a preset format to obtain a synchronous operation message corresponding to at least one data table change operation of the source database.

In one embodiment, the message queue corresponding to the target message topic includes at least one partitioned message queue; the pushing the synchronous operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message theme includes:

determining a partition identifier corresponding to the synchronous operation message under the subject of the target message;

Pushing the synchronous operation message to the at least one partition message queue, and enabling the synchronous operation message to be in the partition message queue corresponding to the partition identification.

In a second aspect, the application further provides a data synchronization device. The device comprises:

the analysis module is used for analyzing the data operation log of the source database to be expanded, and pushing synchronous operation information corresponding to at least one data table change operation of the source database to an information queue corresponding to a target information theme; the target message topic is a message topic corresponding to a data table operated by the data table changing operation;

the information generation module is used for generating consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information comprises mapping relations between data tables in the source database and message consumption configuration information; the message consumption configuration information comprises a message theme and routing information;

the program generating module is used for generating consumer programs subscribed to different message topics according to the consumer program configuration information;

an execution module for executing each of the consumer programs; the consumer program is used for pulling a target synchronous operation message from a message queue corresponding to the subscribed message subject, and executing a data table changing operation matched with the target synchronous operation message in a sub-database of a target database according to the corresponding routing information; the target database is a capacity expansion database corresponding to the source database.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

According to the data synchronization method, the data synchronization device, the computer equipment, the storage medium and the computer program product, the data operation log of the source database to be expanded is analyzed, and the synchronization operation message corresponding to at least one data table change operation is pushed to the message queue corresponding to the target message theme aiming at the at least one data table change operation of the source database; the target message subject is a message subject corresponding to the data table operated by the data table changing operation; then, generating consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information comprises mapping relations between data tables in a source database and message consumption configuration information; the message consumption configuration information comprises a message theme and routing information; then, generating consumer programs subscribing to different message topics according to the consumer program configuration information; then, executing each consumer program; the consumer program is used for pulling the target synchronous operation message from the message queue corresponding to the subscribed message subject, and executing the data table changing operation matched with the target synchronous operation message in the sub-database of the target database according to the corresponding route information; the target database is a capacity expansion database corresponding to the source database.

In this way, because of the mapping relation between each data table and the message consumption configuration information, consumer programs subscribing different message topics can be flexibly generated through the mapping relation, so that the consumer programs can perform data table changing operation matched with the pulled target synchronous operation message on the expansion database corresponding to the source database according to the corresponding route information, the flexible synchronization of data in the source data to the expansion database according to the corresponding message consumption configuration information is realized, the data synchronization during non-system shutdown time is realized, the data importing during system shutdown is not needed, the waiting time of the data synchronization is shortened, and the expansion efficiency of the database is effectively improved.

Drawings

FIG. 1 is a flow chart of a data synchronization method according to an embodiment;

FIG. 2 is a flowchart illustrating steps for generating consumer program configuration information in one embodiment;

FIG. 3 is a schematic diagram of a Kafka producer submitting a synchronize operation message in one embodiment;

FIG. 4 is a flowchart of a data synchronization method according to another embodiment;

FIG. 5 is a schematic flow chart of a DRP data parsing replication and Kafka data synchronization in one embodiment;

FIG. 6 is a block diagram of a data synchronization device in one embodiment;

fig. 7 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

In one embodiment, as shown in fig. 1, a data synchronization method is provided, where the method is applied to a server for illustration, and the server may be implemented by using a separate server or a server cluster formed by a plurality of servers. The method comprises the following steps:

Step S110, analyzing a data operation log of a source database to be expanded, and pushing synchronous operation information corresponding to at least one data table changing operation to an information queue corresponding to a target information theme aiming at least one data table changing operation of the source database.

Wherein each synchronization operation message is assigned its target message topic.

Wherein the target message topic is a message topic (topic) corresponding to the data table operated by the data table change operation.

Wherein the source database may be a distributed system database. For example, but not limited to, a Mysql database.

The data operation log is a bin.log log.

The data table changing operation comprises at least one of INSERT, UPDATE, DELETE and the like.

The data table operated by the data table changing operation is a data table in the source database, and when the source database is a Mysql database, the data table is a Mysql table.

The message queue may be a Kafka message queue. The Kafka is a high-throughput distributed publish-subscribe message system, and the Kafka message queue is mainly used for a scene of high trans-regional delay and simultaneously supports offline data processing and real-time data processing, so that stability and safety of trans-regional data transmission can be ensured by using the Kafka message queue.

Each data table in the source database corresponds to one message theme, namely, the data tables correspond to the message themes one by one.

The synchronization operation message is a message for determining a data table change operation for the source database to indicate a synchronization operation for the capacity expansion database corresponding to the source database.

The synchronization operation message in the present application may also be simply referred to as a message.

In a specific implementation, the server may analyze a data operation log of a source database to be expanded, where the data operation log records an operation log of the MySQL database in a DML (data manipulation language, including, but not limited to INSERT, UPDATE, DELETE) process, so the server may determine a change operation record corresponding to at least one data table change operation of the source database, and generate, according to the change operation record corresponding to the data table change operation, a message corresponding to the data table change operation as a synchronous operation message corresponding to the data table change operation, where, because the data table change operation is performed in each data table of the source database, and each data table corresponds to a message topic, the synchronous operation message corresponding to the data table change operation may be matched with the message topic corresponding to the corresponding data table, and the corresponding message topic will be the target message topic specified by the synchronous operation message, so the server may push the synchronous operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message topic. The target message theme is a message theme corresponding to the data table operated by the data table changing operation.

Step S120, generating the configuration information of the consumer program according to the data table identifiers corresponding to the data tables in the source database.

The consumer program configuration information comprises mapping relations between data tables in the source database and message consumption configuration information.

Wherein the message consumption configuration information includes a message topic and routing information.

In a specific implementation, each data table in the source database can be configured with corresponding routing information, the routing information and the message theme can form message consumption configuration information, in order to establish a mapping relationship between each data table and the message consumption configuration information, the server can acquire a data table identifier corresponding to each data table, and the data table identifier is used for representing the data table, so that the server can establish the mapping relationship between each data table and the message consumption configuration information according to the data table identifier corresponding to each data table, and the mapping relationship is used as consumer program configuration information.

Step S130, generating consumer programs subscribed to different message topics according to the consumer program configuration information.

In a specific implementation, the server may obtain a consumer format template, and generate a consumer program corresponding to each data table according to the consumer program configuration information, that is, according to a mapping relationship between each data table and the message consumption configuration information. Specifically, the message consumption configuration information includes a message theme and routing information, and the server may replace the message theme and routing information in the consumer format template according to the message theme and routing information corresponding to the data tables, so as to generate consumer programs corresponding to the data tables. Because each data table corresponds to a different message topic, consumer programs subscribing to different message topics can be generated.

Step S140, executing each consumer program.

The consumer program is used for pulling the target synchronous operation message from the message queue corresponding to the subscribed message theme.

The consumer program is further used for executing the data table changing operation matched with the target synchronous operation message in the sub-database of the target database according to the corresponding routing information.

The target database is a capacity expansion database corresponding to the source database.

In a specific implementation, the server may execute each consumer program, and the consumer program may pull the synchronization operation message in the message queue corresponding to the subscribed message topic, as the target synchronization operation message corresponding to the consumer program, so that the consumer program may execute, according to the corresponding routing information, the data table change operation matched with the target synchronization operation message in the sub-database of the target database, where the target database is the expansion database corresponding to the source database.

The op_type operation type identifier in the target synchronization operation message may notify the consumer program that the data table corresponding to the target synchronization operation message in the source database changes operation, where the data dictionary of the op_type includes: INSERT, UPDATE, DELETE. The operation type corresponding to the INSERT operation is identified as "op_type": "INSERT". In this way, the consumer program can execute the data table changing operation matched with the target synchronous operation message in the sub-base of the target database.

In practical applications, the server may have at least two kinds of operation logic, one of which is used to execute step S110, and specifically, the server may execute step S110 through a data replication platform (e.g., DRP platform), which is a producer of Kafka. The other running logic is used for executing the steps S120-S140, and the consumer program is the consumer of Kafka.

In the data synchronization method, by analyzing the data operation log of the source database to be expanded, pushing the synchronization operation message corresponding to at least one data table change operation to the message queue corresponding to the target message subject aiming at least one data table change operation of the source database; the target message subject is a message subject corresponding to the data table operated by the data table changing operation; then, generating consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information comprises mapping relations between data tables in a source database and message consumption configuration information; the message consumption configuration information comprises a message theme and routing information; then, generating consumer programs subscribing to different message topics according to the consumer program configuration information; then, executing each consumer program; the consumer program is used for pulling the target synchronous operation message from the message queue corresponding to the subscribed message subject, and executing the data table changing operation matched with the target synchronous operation message in the sub-database of the target database according to the corresponding route information; the target database is a capacity expansion database corresponding to the source database.

In one embodiment, as shown in fig. 2, step S130, generating consumer program configuration information according to the data table identifier corresponding to each data table in the source database, includes:

step S210, data table identifiers corresponding to the data tables in the source database are obtained.

Step S220, a first mapping relation between each data table identifier and each message theme is established, and a second mapping relation between each data table identifier and each routing information is established.

Step S230, generating the configuration information of the consumer program according to the first mapping relation and the second mapping relation.

In practical application, the data table identifier may be a table name corresponding to the data table.

In a specific implementation, in the process of generating the configuration information of the consumer program according to the data table identifiers corresponding to the data tables in the source database, the server can acquire the data table identifiers corresponding to the data tables in the source database, then the server can establish a mapping relationship between the data table identifiers and the message topics as a first mapping relationship, and can also establish a mapping relationship between the data table identifiers and the routing information as a second mapping relationship. In this way, since the message consumption configuration information includes the message theme and the routing information, the server can obtain the mapping relationship between each data table in the source database and the message consumption configuration information according to the first mapping relationship and the second mapping relationship, so as to generate the consumer program configuration information.

In the process of obtaining the data table identifier corresponding to each data table, the server may respond to the sql (Structured Query Language ) instruction to query the data table identifier corresponding to each data table in the source database.

According to the technical scheme, the data table identifiers corresponding to the data tables in the source database are obtained; establishing a first mapping relation between each data table identifier and each message theme, and establishing a second mapping relation between each data table identifier and each routing information; therefore, because the message consumption configuration information comprises the message theme and the routing information, the mapping relation between each data table in the source database and the message consumption configuration information can be accurately obtained according to the first mapping relation and the second mapping relation, and the consumer program configuration information is generated.

In one embodiment, establishing a first mapping relationship between each data table identity and each message topic includes: obtaining a message theme format template; supplementing the data table identifiers corresponding to the data tables to a message topic format template to obtain message topics corresponding to the data tables; and obtaining a first mapping relation according to the message subjects corresponding to the data tables.

In a specific implementation, in a process of establishing a first mapping relationship between each data table identifier and each message topic, the server may acquire a message topic format template, where the message topic format template is a naming specification of the message topic, and the message topic format template may be as follows: "DRP_MYSQL_" +MySQL database server SET alias + "_" +db name + "_" +table name; wherein, the table name represents the data table identifier. And then, supplementing the data table identifiers corresponding to the data tables to the message topic format template to obtain the message topics corresponding to the data tables.

For example, when MySQL database server SET is named "DB0001", the data TABLE is identified as "TABLE1", and the corresponding message subject is "drp_mysql_db0001_dbname_table1"; the data TABLE is identified as "TABLE2" and the corresponding message header is "DRP_MYSQL_DB0001_DBNAME_TABLE2".

Thus, according to the message topics corresponding to the data tables, the server can obtain the first mapping relation between the data table identifiers and the message topics.

According to the technical scheme, a message theme format template is obtained; supplementing the data table identifiers corresponding to the data tables to a message topic format template to obtain message topics corresponding to the data tables, so that the data tables operated by the data table changing operation correspond to different message topics, and the synchronous operation message corresponding to the data table changing operation can be appointed with the message topics corresponding to the data tables operated by the data table changing operation; then, according to the message topics corresponding to the data tables, the first mapping relation between the data table identifications and the message topics can be accurately obtained.

In one embodiment, according to the corresponding routing information, performing a data table changing operation matched with the target synchronous operation message in a sub-database of the target database, including: determining a sub-database of an operation to be executed in a target database according to the corresponding routing information, and taking the sub-database as a target sub-database corresponding to a target synchronous operation message; and executing the data table changing operation matched with the target synchronous operation message in the target sub-library corresponding to the target synchronous operation message.

In a specific implementation, in a process that the consumer program executes the data table changing operation matched with the target synchronous operation message in the sub-database of the target database according to the corresponding routing information, the consumer program corresponding to each data table can determine the sub-database to be operated in the target database according to the corresponding routing information as the target sub-database corresponding to the target synchronous operation message, and then execute the data table changing operation matched with the pulled target synchronous operation message in the target sub-database corresponding to the target synchronous operation message.

According to the technical scheme of the embodiment, the sub-library of the operation to be executed can be accurately determined in the target database according to the corresponding routing information, so that the sub-library is used as the target sub-library corresponding to the target synchronous operation message; then, in the target sub-database corresponding to the target synchronous operation message, the data table changing operation matched with the target synchronous operation message is executed, so that the accurate synchronization of the data in the source database to the sub-database to be stored in the target database can be realized.

In one embodiment, the routing information includes a routing field and a routing algorithm; determining a sub-database of the operation to be executed in the target database according to the corresponding routing information, wherein the sub-database comprises: calculating corresponding routing fields by adopting a corresponding routing algorithm, and determining a database identification; and taking the sub-database corresponding to the sub-database identification in the target database as the sub-database of the operation to be executed.

In a specific implementation, the routing information includes a routing field and a routing algorithm, and in a process of determining a sub-library of an operation to be executed in the target database according to the corresponding routing information, the consumer program may use the corresponding routing algorithm to calculate the corresponding routing field, determine a sub-library identifier, and then use the sub-library corresponding to the sub-library identifier in the target database as the sub-library of the operation to be executed.

In some embodiments, different data tables in the same database support various routing algorithms, and the routing algorithms include, but are not limited to, common routing algorithms such as consistent hash routing algorithms and specified routing algorithms, and also may be custom routing algorithms, that is, the data tables may support various custom routes, so that the routing algorithms corresponding to the data tables may be flexibly configured.

Thus, the consumer program configuration information includes a mapping relationship between each data table in the source database and the message consumption configuration information, the message consumption configuration information includes a message subject and routing information, the routing information includes a routing field and a routing algorithm, and for the convenience of understanding by those skilled in the art, table 1 provides consumer program configuration information:

Table1 consumer program configuration information

For example, if there are 8 sub-databases (SET 1 database, SET2 database, SET3 database, SET4 database, SET5 database, SET6 database, SET7 database and SET8 database) in the target database, the routing field of TABLE1 is routerKey1, and the sub-database is calculated by using the corresponding routing algorithm consistent hash routing algorithm to obtain the sub-database identifier as 5, the SET5 database in the target database is used as the sub-database to be operated, the consumer program performs the data TABLE changing operation matched with the target synchronization operation message in the SET5 database, and other sub-databases do not need to be processed.

According to the technical scheme of the embodiment, the corresponding routing algorithm is adopted to calculate the corresponding routing field, so that the database identification is determined; taking a sub-database corresponding to the sub-database identification in the target database as a sub-database of the operation to be executed; therefore, each data table corresponds to different routing algorithms, corresponding routing fields are calculated according to the routing algorithms, and a sub-base of an operation to be executed, which is matched with a target synchronous operation message generated based on the data table changing operation, can be determined, so that the data table changing operation matched with the target synchronous operation message is executed in the sub-base of the operation to be executed, and the flexible distribution of data in source data to sub-bases in an expansion database according to the corresponding routing algorithms, the routing fields and the message subject is realized.

In one embodiment, analyzing a data operation log of a source database to be expanded, pushing a synchronization operation message corresponding to at least one data table change operation to a message queue corresponding to a target message subject aiming at least one data table change operation of the source database, including: analyzing the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database; and packaging the change operation record into a message in a preset format to obtain a synchronous operation message corresponding to at least one data table change operation of the source database.

In a specific implementation, in a process of analyzing a data operation log of a source database to be expanded and pushing a synchronous operation message corresponding to at least one data table change operation to a message queue corresponding to a target message theme aiming at least one data table change operation of the source database, the server can analyze the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database, and then package the change operation record into a message in a preset format to obtain the synchronous operation message corresponding to at least one data table change operation of the source database.

According to the technical scheme, the change operation record corresponding to at least one data table change operation of the source database can be obtained by analyzing the data operation log; and then, the change operation record is packaged into a message in a preset format, and a synchronous operation message corresponding to at least one data table change operation of the source database is obtained, so that the consumer program can accurately determine the data table change operation executed in the source database through the synchronous operation message.

In one embodiment, the message queues corresponding to the target message topic include at least one partitioned message queue; pushing the synchronous operation message corresponding to the at least one data table changing operation to the message queue corresponding to the target message theme comprises the following steps: determining a partition identifier corresponding to the synchronous operation message under the subject of the target message; pushing synchronous operation information to at least one partition information queue, and pushing the synchronous operation information to the partition information queue corresponding to the partition identification.

In a specific implementation, the Kafka cluster includes one or more Kafka servers, an independent Kafka server is called a broker, and in pushing synchronization operation messages corresponding to at least one data table changing operation to message queues corresponding to target message topics, the broker is used for receiving synchronization operation messages sent by a producer, and routing the synchronization operation messages to the message queues in the broker, so that the Kafka dynamically adds the synchronization operation messages sent by the producer to a disk and gives an offset to each synchronization operation message, and therefore one broker is an example of an application program for the Kafka. In order to balance the load, the message topic is divided into a plurality of partitions (partitions), and each broker stores one or more partitions. Wherein each partition may act as a partition message queue. Thus, the message queue corresponding to the target message topic includes at least one partitioned message queue.

In this way, in the process of pushing the synchronous operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message theme, the server can determine the partition identifier corresponding to the synchronous operation message under the target message theme, send the synchronous operation message to the browser, and route the synchronous operation message to the at least one partition message queue through the browser, and the partition message queue corresponding to the partition identifier.

In practical application, in the process that the data replication platform is used as a Kafka producer, a partition identifier corresponding to a synchronous operation message under a target message subject is determined, the synchronous operation message is sent to a browser, the synchronous operation message is sent to at least one partition message queue through the browser, and the synchronous operation message is submitted by the Kafka producer (producer) for the convenience of understanding of a person skilled in the art in the process of the partition message queue corresponding to the partition identifier. As shown in fig. 3, in the process of sending a synchronization operation message to a browser through a send () method (producer. Send), the synchronization operation message needs to be sent to the browser after a series of actions of a Serializer (sequencer) and a Partitioner (Partitioner). After the synchronization operation message is serialized, the partition to which the synchronization operation message is sent needs to be determined, and if the partition field is specified in the synchronization operation message producer, the role of the partitioner is not needed, because the partition represents the partition identifier corresponding to the partition to which the synchronization operation message is sent. If the partition field is not specified in the synchronous operation message ProducerReord, the partitioner needs to be relied on, and the partitioner calculates the value of the partition according to the key field to obtain the partition identification. The role of the partitioner is to allocate partitions for synchronous operation messages. Thus, after determining the partition identifier corresponding to the synchronous operation message under the subject of the target message, the synchronous operation message is sent to the browser. In case that the writing of the synchronous operation message is unsuccessful, the Broker allocates partitions for the synchronous operation message to perform retries.

According to the technical scheme of the embodiment, the message queues corresponding to the target message subject comprise at least one partition message queue; determining partition identifiers corresponding to the synchronous operation messages under the subject of the target message; pushing synchronous operation information to at least one partition information queue, and pushing synchronous operation information to the partition information queue corresponding to the partition identification; thus, by distributing different synchronous operation messages to different partition message queues, load balancing is facilitated.

In another embodiment, as shown in fig. 4, a data synchronization method is provided, and the method is applied to a server for illustration, and includes the following steps:

step S410, a data operation log of a source database to be expanded is analyzed, and a change operation record corresponding to at least one data table change operation of the source database is obtained.

Step S420, the change operation record is packaged into a message with a preset format, and a synchronous operation message corresponding to at least one data table change operation of the source database is obtained.

Step S430, pushing at least one synchronous operation message corresponding to the data table changing operation to a message queue corresponding to the target message theme.

Step S440, data table identifiers corresponding to the data tables in the source database are obtained.

Step S450, a first mapping relation between each data table identifier and each message theme is established, and a second mapping relation between each data table identifier and each routing information is established.

Step S460, generating the configuration information of the consumer program according to the first mapping relation and the second mapping relation.

In step S470, consumer programs subscribing to different message topics are generated based on the consumer program configuration information.

Step S480, executing each consumer program.

It should be noted that, the specific limitation of the above steps may be referred to the specific limitation of a data synchronization method.

In some embodiments, for ease of understanding by those skilled in the art, fig. 5 provides a schematic flow diagram of data parsing replication of a DRP (data replication platform) and Kafka data synchronization in the case of data synchronization using a DRP platform and Kafka message queues. As shown in fig. 5, the parsing service and the garage-falling service are operated in the server of the above embodiment, and the parsing service corresponds to the producer of Kafka; the library-dropping service is used for generating a consumer program, and the library-dropping service is equivalent to a consumer of Kafka; the message engine cluster is a Kafka cluster. The monitoring service corresponding to the analysis service is used for monitoring data abnormality in the operation process of the analysis service, and the monitoring service corresponding to the database-falling service is used for monitoring data abnormality in the operation process of the database-falling service.

In practical applications, the parsing service and the library-dropping service may also be respectively run in different servers, which is not limited herein.

In some embodiments, a Kafka cluster generally includes multiple agents. The Kafka agents were managed and coordinated using a ZooKeeper. Each Kafka agent coordinates the other Kafka agents through a ZooKeeper. The ZooKeeper service will notify the producer and consumer when an agent is newly added to the Kafka system or if a certain agent fails. The producer and consumer accordingly begin to coordinate with other agents.

Specifically, the role of the zookeeper is that the producer can push the synchronous operation message to each agent, when one agent fails, the zookeeper informs the producer that the synchronous operation message is not needed to be sent to the failed agent, and the consumer does not receive the synchronous operation message from the failed agent instead of receiving the synchronous operation message from other normal agents. If insufficient proxy resources are found in the capacity expansion process, a large amount of data cannot be rapidly synchronized, for example, only 4 proxies can be added, 8 proxies are added after the 4 proxies are added, a producer is notified of the availability of new proxies by the zookeeper, synchronous operation information can be pushed to the new proxies simultaneously, a consumer obtains the synchronous operation information from one of the original 4 proxies, the synchronous operation information can be obtained from one of the 8 proxies, and the new proxies share the load pressure.

To balance the load, topics are divided into multiple partitions, with each agent storing one or more partitions. Multiple producers and consumers can simultaneously produce and acquire synchronized operation messages. Kafka's Topic (message Topic) can be logically considered as a queue of messages, each consumption must specify its Topic, and it can be simply understood that it must specify into which queue this synchronization message is placed. In order to allow the throughput rate of Kafka to be increased linearly, topic is physically divided into one or more partitions, each of which physically corresponds to a folder under which all of the synchronization operation messages and index files for that Partition are stored.

In order to ensure the correctness of the data, iteration can be performed on all synchronous operation messages in the latest log segment so as to verify the validity of each message item. A message item is legal only if its size plus offset is less than the file size and the CRC32 value of the payload in the message is equal to the CRC value stored in the message. In case of data corruption detected, the file is truncated by the last valid offset. The message consumer can rewind back to any location to re-consume, and when the consumer fails, the minimum offset can be selected to re-read the consumed message.

In order to ensure the reliability of message delivery, in practical application, the corresponding message delivery modes can be selected according to different requirements, and generally three types of message delivery modes are as follows:

the At most once message may be lost but never retransmitted

The At least one message is never lost, but may be retransmitted

Exact once each message must be transmitted once and only once, which is often desired by the user.

As described above, the technical solution of the embodiment adopts the technology based on DRP data analysis replication and Kafka data synchronization to realize that the distributed system supports flexible routing algorithm fragmentation expansion, so that the problems that the expansion time consumption of the distributed system using Mysql database with large data magnitude is long, and the system needs to be backed back integrally due to unexpected situations, so that the emergency back-off data backup requirement cannot be met can be solved.

Wherein Kafka provides message persistence capability in a manner that the time complexity is O (1), and can ensure access performance of constant time complexity even for data above TB level. The DRP and the Kafka can start data synchronization and cleaning in non-stop time without waiting time for data import and cleaning to calculate data synchronization. Furthermore, the Kafka client supports most of the current mainstream languages, including: C. c++, erlang, java,. Net, perl, PHP, python, ruby, go, javascript, can communicate with the Kafka server using any of the above languages.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data synchronization device for realizing the above related data synchronization method. The implementation of the solution provided by the apparatus is similar to that described in the above method, so the specific limitation of one or more embodiments of the data synchronization apparatus provided below may be referred to as the limitation of a data synchronization method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 6, there is provided a data synchronization apparatus including: a parsing module 610, an information generation module 620, a program generation module 630, and an execution module 640, wherein:

the parsing module 610 is configured to parse a data operation log of a source database to be expanded, and push, for at least one data table change operation of the source database, a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic; the target message topic is a message topic corresponding to a data table operated by the data table changing operation.

The information generating module 620 is configured to generate consumer program configuration information according to the data table identifiers corresponding to the data tables in the source database; the consumer program configuration information comprises mapping relations between data tables in the source database and message consumption configuration information; the message consumption configuration information includes a message topic and routing information.

Program generation module 630 is configured to generate, according to the consumer program configuration information, consumer programs subscribed to different message topics.

An execution module 640 for executing each of the consumer programs; the consumer program is used for pulling a target synchronous operation message from a message queue corresponding to the subscribed message subject, and executing a data table changing operation matched with the target synchronous operation message in a sub-database of a target database according to the corresponding routing information; the target database is a capacity expansion database corresponding to the source database.

In one embodiment, the information generating module 620 is specifically configured to obtain a data table identifier corresponding to each data table in the source database; establishing a first mapping relation between each data table identifier and each message theme, and establishing a second mapping relation between each data table identifier and each routing information; and generating the consumer program configuration information according to the first mapping relation and the second mapping relation.

In one embodiment, the information generating module 620 is specifically configured to obtain a message topic format template; supplementing the data table identifiers corresponding to the data tables to the message topic format template to obtain the message topics corresponding to the data tables; and obtaining the first mapping relation according to the message subjects corresponding to the data tables.

In one embodiment, the executing module 640 is specifically configured to determine, according to the corresponding routing information, a sub-library of an operation to be executed in the target database, as a target sub-library corresponding to the target synchronous operation message; and executing the data table changing operation matched with the target synchronous operation message in the target sub-library corresponding to the target synchronous operation message.

In one embodiment, the routing information includes a routing field and a routing algorithm; the execution module 640 is specifically configured to calculate a corresponding routing field by using a corresponding routing algorithm, and determine a database identifier; and taking the sub-database corresponding to the sub-database identifier in the target database as the sub-database of the operation to be executed.

In one embodiment, the parsing module 610 is specifically configured to parse the data operation log to obtain a change operation record corresponding to at least one data table change operation of the source database; and packaging the change operation record into a message in a preset format to obtain a synchronous operation message corresponding to at least one data table change operation of the source database.

In one embodiment, the message queue corresponding to the target message topic includes at least one partitioned message queue; the parsing module 610 is specifically configured to determine a partition identifier corresponding to the synchronization operation message under the target message subject; pushing the synchronous operation message to the at least one partition message queue, and enabling the synchronous operation message to be in the partition message queue corresponding to the partition identification.

The modules in the data synchronization device may be implemented in whole or in part by software, hardware, or a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing consumer program configuration information data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data synchronization method.

It will be appreciated by those skilled in the art that the structure shown in FIG. 7 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of data synchronization, the method comprising:

2. The method of claim 1, wherein generating the consumer program configuration information based on the table identifier corresponding to each table in the source database comprises:

3. The method of claim 2, wherein said establishing a first mapping between each of said data table identifications and each of said message topics comprises:

obtaining a message theme format template;

4. The method of claim 1, wherein the performing a data table change operation in a database of a target database that matches the target synchronization operation message according to the corresponding routing information comprises:

5. The method of claim 4, wherein the routing information includes a routing field and a routing algorithm; the determining the sub-database of the operation to be executed in the target database according to the corresponding routing information comprises the following steps:

6. The method according to claim 1, wherein the parsing the data operation log of the source database to be expanded, for at least one data table change operation of the source database, pushing a synchronization operation message corresponding to the at least one data table change operation to a message queue corresponding to a target message topic, includes:

7. The method of claim 1, wherein the message queue corresponding to the target message topic comprises at least one partitioned message queue; the pushing the synchronous operation message corresponding to the at least one data table change operation to the message queue corresponding to the target message theme includes:

8. A data synchronization device, the device comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.