CN112948494A

CN112948494A - Data synchronization method and device, electronic equipment and computer readable medium

Info

Publication number: CN112948494A
Application number: CN202110242006.XA
Authority: CN
Inventors: 孟可; 彭安; 张谦
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-03-04
Filing date: 2021-03-04
Publication date: 2021-06-11

Abstract

The embodiment of the disclosure discloses a data synchronization method, a data synchronization device, an electronic device and a computer readable medium. One embodiment of the method comprises: receiving data synchronization task type information in response to successful connection with the target database; acquiring data to be synchronized corresponding to the data synchronization task type information from the target database; and synchronizing the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a mode of synchronizing the data to be synchronized to the backup database. The embodiment improves the data synchronization efficiency and reduces the waste of bandwidth resources.

Description

Data synchronization method and device, electronic equipment and computer readable medium

Technical Field

Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a data synchronization method, apparatus, electronic device, and computer-readable medium.

Background

With the popularization and development of computer technology, data generated by network application is increased in a blowout manner. The prior art often adopts a distributed database to store mass data. In order to avoid data loss or damage caused when the distributed database fails, data in the database to be backed up is often required to be synchronized to the backup database. At present, the commonly adopted mode is as follows: and acquiring the data to be backed up from the database to be backed up, and synchronizing the data to be backed up to the backup database.

However, when the data synchronization is performed in the above manner, there are often the following technical problems:

often, a data synchronization mode matched with the data synchronization scene is not selected according to the data synchronization scene, so that the data synchronization efficiency is low, and further, the waste of bandwidth resources is caused.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Some embodiments of the present disclosure propose a data synchronization method, apparatus, electronic device and computer readable medium to solve one of the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a data synchronization method, including: receiving data synchronization task type information in response to successful connection with the target database; acquiring data to be synchronized corresponding to the data synchronization task type information from the target database; and synchronizing the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a mode of synchronizing the data to be synchronized to the backup database.

Optionally, the obtaining data to be synchronized corresponding to the data synchronization task type information from the target database includes: and acquiring full data from the target database as the data to be synchronized in response to the fact that the data synchronization task type information is determined to be a first data synchronization type.

Optionally, the obtaining data to be synchronized corresponding to the data synchronization task type information from the target database includes: and acquiring incremental data from the target database as data to be synchronized in response to the fact that the data synchronization task type information is determined to be a second data synchronization type.

Optionally, the obtaining data to be synchronized corresponding to the data synchronization task type information from the target database includes: and acquiring full data and incremental data from the target database as the data to be synchronized in response to the determination that the data synchronization task type information is a third data synchronization type.

Optionally, the synchronizing the data to be synchronized to the backup database according to the concurrent execution type includes: and in response to determining that the concurrent execution type is a first concurrent execution type, synchronizing each record in the data to be synchronized to the backup database in a serial manner.

Optionally, the synchronizing the data to be synchronized to the backup database according to the concurrent execution type includes: in response to determining that the concurrent execution type is a second concurrent execution type, generating at least one database operation statement based on each record in the data to be synchronized; and executing each database operation statement in the at least one database operation statement.

Optionally, the at least one database statement includes: at least one database insert statement; and generating at least one database operation statement based on each record in the data to be synchronized, including: and in response to the fact that the data to be synchronized comprises full data, converting each record in the full data into a database insert statement to obtain at least one database insert statement.

Optionally, the at least one database statement further includes: at least one sub-database operation statement; and generating at least one database operation statement based on each record in the data to be synchronized, further comprising: in response to the fact that the data to be synchronized comprises incremental data, generating candidate sub-database operation statements according to the event type corresponding to each record in the incremental data to obtain at least one candidate sub-database operation statement; and in response to determining that a target data table meets a first parallel processing condition, merging candidate sub-database operation statements meeting a merging condition in the at least one candidate sub-database operation statement to generate a sub-database operation statement, and obtaining the at least one sub-database operation statement, wherein the target data table is a data table corresponding to the data to be synchronized, the first parallel processing condition is that the target data table comprises a unique index or a main key, and the merging condition is that the main key or the unique index of the data tables operated by the candidate sub-database operation statements is the same.

Optionally, the executing each database operation statement in the at least one database operation statement includes: and executing each database operation statement in the at least one database operation statement in a serial working mode in response to determining that the target data table meets a second parallel processing condition, wherein the second parallel processing condition is that the target data table contains a unique index and a main key.

Optionally, the synchronizing the data to be synchronized to the backup database according to the concurrent execution type includes: in response to determining that the concurrent execution type is a third concurrent execution type, performing transaction analysis on the data to be synchronized to generate at least one transaction to be processed; determining a record information group corresponding to each transaction to be processed in the at least one transaction to be processed to obtain a record information group set; selecting a preset number of to-be-processed transactions from the at least one to-be-processed transaction as target to-be-processed transactions to obtain a target to-be-processed transaction set; performing transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information and obtain a transaction conflict detection information set; generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set, wherein nodes in the directed acyclic graph correspond to target to-be-processed transactions in the target to-be-processed transaction set one to one; carrying out topological sorting on the directed acyclic graph to generate a topological sorting sequence; and executing the target to-be-processed transaction corresponding to each node in the topology sequencing sequence.

In a second aspect, some embodiments of the present disclosure provide a data synchronization apparatus, including: a receiving unit configured to receive data synchronization task type information in response to a successful connection with a target database; an obtaining unit configured to obtain data to be synchronized corresponding to the data synchronization task type information from the target database; and the synchronization unit is configured to synchronize the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a mode of synchronizing the data to be synchronized to the backup database.

Optionally, the obtaining unit is further configured to: and acquiring full data from the target database as the data to be synchronized in response to the fact that the data synchronization task type information is determined to be a first data synchronization type.

Optionally, the obtaining unit is further configured to: and acquiring incremental data from the target database as data to be synchronized in response to the fact that the data synchronization task type information is determined to be a second data synchronization type.

Optionally, the obtaining unit is further configured to: and acquiring full data and incremental data from the target database as the data to be synchronized in response to the determination that the data synchronization task type information is a third data synchronization type.

Optionally, the synchronization unit is further configured to: and in response to determining that the concurrent execution type is a first concurrent execution type, synchronizing each record in the data to be synchronized to the backup database in a serial manner.

Optionally, the synchronization unit is further configured to: in response to determining that the concurrent execution type is a second concurrent execution type, generating at least one database operation statement based on each record in the data to be synchronized; and executing each database operation statement in the at least one database operation statement.

Optionally, the at least one database statement includes: at least one database insert statement; and the synchronization unit is further configured to: and in response to the fact that the data to be synchronized comprises full data, converting each record in the full data into a database insert statement to obtain at least one database insert statement.

Optionally, the at least one database statement further includes: at least one sub-database operation statement; and the synchronization unit is further configured to: in response to the fact that the data to be synchronized comprises incremental data, generating candidate sub-database operation statements according to the event type corresponding to each record in the incremental data to obtain at least one candidate sub-database operation statement; and in response to determining that a target data table meets a first parallel processing condition, merging candidate sub-database operation statements meeting a merging condition in the at least one candidate sub-database operation statement to generate a sub-database operation statement, and obtaining the at least one sub-database operation statement, wherein the target data table is a data table corresponding to the data to be synchronized, the first parallel processing condition is that the target data table comprises a unique index or a main key, and the merging condition is that the main key or the unique index of the data tables operated by the candidate sub-database operation statements is the same.

Optionally, the synchronization unit is further configured to: and executing each database operation statement in the at least one database operation statement in a serial working mode in response to determining that the target data table meets a second parallel processing condition, wherein the second parallel processing condition is that the target data table contains a unique index and a main key.

Optionally, the synchronization unit is further configured to: in response to determining that the concurrent execution type is a third concurrent execution type, performing transaction analysis on the data to be synchronized to generate at least one transaction to be processed; determining a record information group corresponding to each transaction to be processed in the at least one transaction to be processed to obtain a record information group set; selecting a preset number of to-be-processed transactions from the at least one to-be-processed transaction as target to-be-processed transactions to obtain a target to-be-processed transaction set; performing transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information and obtain a transaction conflict detection information set; generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set, wherein nodes in the directed acyclic graph correspond to target to-be-processed transactions in the target to-be-processed transaction set one to one; carrying out topological sorting on the directed acyclic graph to generate a topological sorting sequence; and executing the target to-be-processed transaction corresponding to each node in the topology sequencing sequence.

In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first aspect.

In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium on which a computer program is stored, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect.

The above embodiments of the present disclosure have the following beneficial effects: according to the data synchronization method of some embodiments of the disclosure, data synchronization tasks under various data synchronization scenes are met, and data synchronization efficiency is improved. Specifically, the reasons for the low efficiency of data synchronization are: often, a data synchronization mode matched with the data synchronization scene cannot be selected according to the data synchronization scene. Based on this, the data synchronization method of some embodiments of the present disclosure first receives data synchronization task type information in response to a successful connection with a target database. In practical situations, when data in the target database is synchronously backed up, it is often necessary to ensure that the target database and the backup database are successfully connected in communication. And when the connection is successful, the data synchronization task type information is acquired, so that the condition of data synchronization failure caused by connection failure is avoided. And secondly, acquiring data to be synchronized corresponding to the data synchronization task type information from the target database. According to the data synchronization task type information, the corresponding data to be synchronized is obtained, and the data synchronization pressure caused by directly synchronizing all data in the target database is reduced. And finally, synchronizing the data to be synchronized to a backup database according to the concurrent execution type. In practical situations, the amount of data to be synchronized by different data synchronization tasks is often different. And the data synchronization mode is flexibly selected through the concurrent execution type, so that the data synchronization efficiency is improved, and the waste of bandwidth resources is reduced.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 is a schematic diagram of one application scenario of a data synchronization method of some embodiments of the present disclosure;

FIG. 2 is a flow diagram of some embodiments of a data synchronization method according to the present disclosure;

FIG. 3 is a flow diagram of further embodiments of a data synchronization method according to the present disclosure;

FIG. 4 is a schematic diagram of a target graph in the data synchronization method of the present disclosure;

FIG. 5 is a schematic diagram of a directed acyclic graph in the data synchronization method of the present disclosure;

FIG. 6 is a schematic diagram of a topological sort sequence in the data synchronization method of the present disclosure;

FIG. 7 is a schematic block diagram of some embodiments of a data synchronization apparatus according to the present disclosure;

FIG. 8 is a schematic structural diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 is a schematic diagram of an application scenario of a data synchronization method of some embodiments of the present disclosure.

In the application scenario of fig. 1, first, the computing device 101 may receive data synchronization task type information 103 in response to a successful connection with the target database 102 (e.g., the data synchronization task type information 103 may represent a multilist synchronization type). Then, the computing device 101 may obtain the data to be synchronized 104 corresponding to the data synchronization task type information 103 from the target database 102. Finally, the computing device 101 may synchronize the aforementioned data to be synchronized 104 to the backup database 106 according to a concurrent execution type 105 (e.g., an "asynchronous copy" type). The backup database 106 is used for backing up data in the target database 102. The concurrent execution type 105 may be used to characterize the manner in which the data to be synchronized 104 is synchronized to the backup database 106.

The computing device 101 may be hardware or software. When the computing device is hardware, it may be implemented as a distributed cluster composed of multiple servers or terminal devices, or may be implemented as a single server or a single terminal device. When the computing device is embodied as software, it may be installed in the hardware devices enumerated above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.

It should be understood that the number of computing devices in FIG. 1 is merely illustrative. There may be any number of computing devices, as implementation needs dictate.

With continued reference to fig. 2, a flow 200 of some embodiments of a data synchronization method according to the present disclosure is shown. The data synchronization method comprises the following steps:

step 201, in response to successful connection with the target database, receiving data synchronization task type information.

In some embodiments, an executing agent of the data synchronization method (e.g., computing device 101 shown in fig. 1) may receive data synchronization task type information via a wired connection or a wireless connection in response to a successful connection with the target database. The target database may be a database that needs to perform data synchronization. The target database may be a distributed database. The target database may be a non-distributed database. For example, the target database may be a MongoDB database. The target database may also be a MySQL database. The data synchronization task type information can be used for characterizing the type of the data synchronization task. The data synchronization task type information may be manually configured. The data sync task type information may be a "single table sync type". The data synchronization task type information may be a "multilist synchronization type". The "single table synchronization type" may be characterized in that data synchronization is performed on data in the target database in units of one data table. The "multi-table synchronization type" may be characterized in that data synchronization is performed on data in the target database by taking a target number of data tables as a unit. The execution subject may be responsive to receiving a database connection success signal to determine that the connection with the target database was successful. The database connection success signal may be "200". The "200" may characterize a successful connection to the target database. The data synchronization scene can represent the task type of data synchronization.

As an example, the above data synchronization scenario may be a "single table synchronization type". The data synchronization scenario described above may also be of the "multilist synchronization type".

Optionally, the data synchronization task type information may also be one item not limited to the following: the "first data sync type", "second data sync type", and "third data sync type". The "first data synchronization type" may represent data synchronization of the total amount of data in the target database. The "second data synchronization type" may characterize data synchronization of the incremental data in the target database. The "third data synchronization type" may be a data synchronization characterizing the full amount of data and the incremental data in the target database. The full amount of data may refer to all data stored in the target database. The incremental data may refer to new data stored in the target database after the previous data synchronization.

As yet another example, the above data synchronization scenario may be a "first data synchronization type". The above-described data synchronization scenario may also be a "second data synchronization type". The above-described data synchronization scenario may also be a "third data synchronization type".

Step 202, acquiring data to be synchronized corresponding to the data synchronization task type information from the target database.

In some embodiments, the acquiring, by the executing entity, data to be synchronized corresponding to the data synchronization task type information from the target database may include:

firstly, table information of all data tables stored in the target database is obtained, and a table information sequence is obtained.

The table information in the table information sequence may include: data table name.

As an example, the execution body may obtain the table information of all the data tables stored in the target database through the following SQL (Structured Query Language) statement:

SELECT TALBE_NAME

FROM information_schema."TABLES"

WHERE TABLE_SCHEMA＝"db_name"。

wherein, the db _ name may be a name of the target database.

As yet another example, the above-described table information sequence may be [ "table 1", "table 2", "table 3", "table 4" ].

And secondly, acquiring data to be synchronized corresponding to the data synchronization task type information from the target database through an SQL statement according to the data synchronization task type information and the table information sequence.

The data to be synchronized may represent data that needs to be synchronized in the target database. The data to be synchronized may be at least one line of data in the target database. The data to be synchronized may also be at least one piece of transaction information in a binary log file corresponding to the target database. The binary log file may be a MySQLBinlog log file. The transaction corresponding to the transaction information may be a database transaction.

When the data synchronization task type information is a single table synchronization type. First, the execution body may generate a first query statement according to each table information in the table information sequence, so as to obtain a first query statement sequence. Then, the data to be synchronized may be obtained according to each first query statement in the first query statement sequence.

When the data synchronization task type information is a multi-table synchronization type. First, the execution body may generate a second query statement from each table information in the table information sequence. Then, the data to be synchronized may be acquired according to the second query statement. The first query statement in the first query statement sequence may be an "INSERT" type statement in an SQL statement. The second query statement may be an "INSERT" type statement in an SQL statement. As an example, the above table information sequence may be [ "table 1", "table 2", "table 3", "table 4" ]. The data sync task type information may be a "single table sync type". The first query statement may be "SELECT FROM table 1". The execution body may obtain the data to be synchronized through "SELECT FROM table 1". The above-mentioned data to be synchronized may be "[ record 1, record 2 ]".

As yet another example, the data sync task type information may be a "multi-table sync type". The target number may be "2". The above table information sequence may be [ "table 1", "table 2", "table 3", "table 4" ]. The second query statement may be "SELECT FROM table1, table 2". The execution body may obtain the data to be synchronized through "SELECT FROM table1, table 2". The above-mentioned data to be synchronized may be "[ record 1, record 2, record 3, record 4 ]".

In some optional implementation manners of some embodiments, the execution subject may, in response to determining that the data synchronization task type information is a first data synchronization type, obtain, through an SQL statement, full data from the target database as the data to be synchronized.

As an example, the execution body may obtain the full data as the data to be synchronized through an SQL statement "SELECT × FROM data table name".

In some optional implementation manners of some embodiments, the executing entity may, in response to determining that the data synchronization task type information is of the second data synchronization type, obtain incremental data from the target database as data to be synchronized.

The execution main body can acquire incremental data from the target database as data to be synchronized through SQL statements according to the first target time and the second target time. The first target time may be a time when the previous data synchronization is completed. The second target time may be a current time.

As an example, the execution body may obtain incremental data from the target database as the data to be synchronized through the following SQL statements:

SELECT*

FROM data table name

The WHERE timestamp BETWEEN is the first target time AND the second target time.

In some optional implementation manners of some embodiments, in response to determining that the data synchronization task type information is a third data synchronization type, the executing entity may obtain, as the data to be synchronized, full data and incremental data from the target database by:

firstly, acquiring the full data from the target database through SQL statements.

As an example, the execution body may obtain the full data through an SQL statement "SELECT × FROM data table name".

And secondly, acquiring all records stored in the temporary data table as incremental data.

The execution body may create a trigger, and when a new record is generated, the trigger may store the new record to the temporary data table while storing the new record to the target database. When the execution main body obtains the incremental data of the target database, the execution main body may directly determine the record stored in the temporary data table as the incremental data. When the data volume of the target database is large, the reading and writing frequency of the target database can be reduced in such a way, and the reading and writing pressure of the target database is reduced.

And step 203, synchronizing the data to be synchronized to a backup database according to the concurrent execution type.

In some embodiments, the execution main body may synchronize data to be synchronized to the backup database by a wired connection or a wireless connection according to a concurrent execution type. The concurrent execution type may represent a manner of synchronizing the data to be synchronized to the backup database. The backup database may be a database for backing up data in the target database. The backup database may be a distributed database. The backup database may be a non-distributed database. For example, the backup database may be a MongoDB database. The backup database may also be a MySQL database. The concurrent execution type can be generated by means of manual configuration. The execution main body may also determine a concurrent execution type according to the data amount of the data to be synchronized. The concurrent execution type may be an "asynchronous copy" type. The concurrent execution type may also be a "synchronous replication" type. When the data amount of the data to be synchronized is smaller than the predetermined size, the execution main body may adopt a "synchronous copy" type to synchronize the data to be synchronized to the backup database. When the data amount of the data to be synchronized is greater than or equal to the predetermined size, the execution main body may adopt an "asynchronous copy" type to synchronize the data to be synchronized to a backup database. The predetermined size may be 200 megabits. Here, the setting of the predetermined size is not limited.

The above embodiments of the present disclosure have the following beneficial effects: according to the data synchronization method of some embodiments of the disclosure, data synchronization tasks under various data synchronization scenes are met, and data synchronization efficiency is improved. Specifically, the reasons for the low efficiency of data synchronization are: often, a data synchronization mode matched with the data synchronization scene cannot be selected according to the data synchronization scene. Based on this, the data synchronization method of some embodiments of the present disclosure first receives data synchronization task type information in response to a successful connection with a target database. In practical situations, when data in the target database is synchronously backed up, it is often necessary to ensure that the target database and the backup database are successfully connected in communication. And when the connection is successful, the data synchronization task type information is acquired, so that the condition of data synchronization failure caused by connection failure is avoided. And secondly, acquiring data to be synchronized corresponding to the data synchronization task type information from the target database. According to the data synchronization task type information, the corresponding data to be synchronized is obtained, and the data synchronization pressure caused by directly synchronizing all data in the target database is reduced. And finally, synchronizing the data to be synchronized to a backup database according to the concurrent execution type. In practical situations, different data synchronization tasks often require different amounts of data to be synchronized. And through the concurrent execution type, a data synchronization mode is flexibly selected, the data synchronization efficiency is improved, and the consumption of bandwidth resources is reduced.

With further reference to FIG. 3, a flow 300 of further embodiments of a data synchronization method is illustrated. The process 300 of the data synchronization method includes the following steps:

step 301, in response to a successful connection with the target database, receiving data synchronization task type information.

Step 302, obtaining data to be synchronized corresponding to the data synchronization task type information from the target database.

In some embodiments, the specific implementation of

steps

301 and 302 and the technical effects thereof can refer to

steps

201 and 202 in the embodiments corresponding to fig. 2, which are not described herein again.

Step 303, in response to determining that the concurrent execution type is the first concurrent execution type, synchronizing each record in the data to be synchronized to the backup database in a serial manner.

In some embodiments, an execution subject of the data synchronization method (e.g., the computing device 101 shown in fig. 1) may synchronize each record in the data to be synchronized to the backup database in a serial manner through a wired connection or a wireless connection in response to determining that the concurrent execution type is the first concurrent execution type. The first concurrent execution type may be characterized by a serial manner, and perform data synchronization on the data to be synchronized.

Step 304, in response to determining that the concurrent execution type is the second concurrent execution type, generating at least one database operation statement based on each record in the data to be synchronized; and executing each database operation statement in the at least one database operation statement.

In some embodiments, the execution subject may generate at least one database operation statement based on each record in the data to be synchronized, in response to determining that the concurrent execution type is the second concurrent execution type; and executing each database operation statement in the at least one database operation statement. The database operation statement may be an SQL statement for updating records in the database. The database operation statement may be an "UPDATE statement" in an SQL statement.

Optionally, the at least one database statement may include: at least one database insert statement. Optionally, the at least one database statement may further include: at least one sub-database operation statement. The database insertion statement in the at least one database insertion statement may be an SQL statement for adding records to a database. For example, the database insertion statement may be an "INSERT" statement in an SQL statement. The second concurrent execution type may be characterized by taking a data table as a unit, and performing data synchronization on the data to be synchronized in a concurrent manner.

The execution main body may fill each piece of data in each record in the data to be synchronized into a preset database operation statement to generate a database operation statement, so as to obtain the at least one database operation statement.

First, the execution body may add the at least one database operation statement to a target file. The target file may be a file with sql as a suffix. Then, the execution agent may execute a "source name of the target file sql" command. To enable execution of each of the at least one database operation statement.

Optionally, the execution main body may convert each record in the full amount of data into a database insertion statement to obtain at least one database insertion statement in response to determining that the data to be synchronized includes the full amount of data. The execution main body can splice each record in the full data with the preset database operation statement so as to convert each record in the full data into a database insertion statement.

As an example, the record in the above data to be synchronized may be "0001, zhang san, man, 23 years old". The preset database operation statement may be "backup data table name VALUES ()" in the INSERT INTOs backup database. The database INSERT statement corresponding to the record "0001, zhang san, man, 23 year old" may be "backup data table name VALUES in INSERT INTOs backup database (0001, zhang san, man, 23 year old)".

Optionally, the generating, by the execution main body, at least one database operation statement based on each record in the data to be synchronized may include:

step one, in response to the fact that the data to be synchronized comprises incremental data, generating candidate sub-database operation statements according to an event type corresponding to each record in the incremental data, and obtaining at least one candidate sub-database operation statement.

Wherein, the event type can characterize the operation type. The event type may be an "update operation" type. The event type described above may also be an "insert operation type". The event type described above may also be an "update operation" type.

As an example, the incremental data described above may include records of [ "0002, zhangsi, male, 22 years old", "0002, zhangwu, female, 21 years old", "0004, zhangliu, male, 24 years old", "0004, zhangliu, male, 23 years old" ]. The type of the event corresponding to "0002, zhangsi, male, and 22 years old" may be "update operation type". The generated candidate sub-database operation statement may be "UPDATE data table 1SET attribute 1 ═ zhangsi", "attribute 2 ═ man", and attribute 3 ═ 22 year "WHERE attribute 4 (primary key) ═ 0002" ".

And secondly, in response to the fact that the target data table meets the first parallel processing condition, combining the candidate sub-database operation statements meeting the combination condition in the at least one candidate sub-database operation statement to generate a sub-database operation statement, and obtaining the at least one sub-database operation statement.

The target data table may be a data table corresponding to the data to be synchronized. The first parallel processing condition may be "the target data table includes a unique index or a primary key". The merge condition may be "the primary key or unique index of the data table operated by the candidate sub-database operation statement is the same".

As an example, the candidate sub-database operation statement may be "UPDATE data table 1SET attribute 1 ═ zhangsi", "attribute 2 ═ man", and attribute 3 ═ 22 year "WHERE attribute 4 (primary key) ═ 0002" ". The candidate sub-database operation statement may also be "UPDATE data table 1SET attribute 1 ═ piece five," attribute 2 ═ piece female, "attribute 3 ═ piece 21" WHERE attribute 4 (primary key) ═ piece 0002 "". The illustrated candidate sub-database operation statements satisfy the merge condition. The execution body may merge the illustrated candidate sub-database operation statements in response to determining that the target data table satisfies the first parallel processing condition. The sub-database operation statement generated after merging may be "UPDATE data table 1SET attribute 1 ═ piece five", attribute 2 ═ female ", attribute 3 ═ 21 year" WHERE attribute 4 (primary key) ═ 0002 "".

As yet another example, the execution body may merge the candidate sub-database operation statement "INSERT data table2 intovalues (0004, six, male, 24 years old)" and the candidate sub-database operation statement "update data table 2SET attribute 1 ═ four" by "attribute 2 ═ male" attribute 3 ═ 22 year old "WHERE attribute 4 (primary key) ═ 0002" ". The sub-database operation statement generated after merging may be "INSERT data table 2INTO VALUES (0004, zhang liu, man, 23 years)".

Optionally, each of the at least one database operation statement is executed in a serial working manner in response to determining that the target data table satisfies the second parallel processing condition.

The second parallel processing condition is that "the target data table includes a unique index and includes a primary key". The primary key may be a value for uniquely identifying a record in a data table.

As an example, the record may be "0001, zhang san, male". Where "0001" may be the primary key of the above record.

Step 305, in response to determining that the concurrent execution type is the third concurrent execution type, performing transaction analysis on the data to be synchronized to generate at least one transaction to be processed.

In some embodiments, the execution subject may perform transaction parsing on the data to be synchronized to generate at least one transaction to be processed in response to determining that the concurrent execution type is the third concurrent execution type. The execution main body may generate a to-be-processed transaction according to each piece of transaction information in at least one piece of transaction information included in the to-be-synchronized data, so as to obtain the at least one to-be-processed transaction. The pending transaction of the at least one pending transaction may be a database transaction. A database transaction may characterize a sequence of database operations that access and possibly operate on various data items. The third concurrent execution type may be characterized by a transaction bit for performing data synchronization on the data to be synchronized.

Step 306, determining a record information group corresponding to each to-be-processed transaction in at least one to-be-processed transaction, and obtaining a record information group set.

In some embodiments, the executing entity may determine at least one record operated by the to-be-processed transaction as the record information group corresponding to the to-be-processed transaction.

As an example, the SQL statement corresponding to the to-be-processed transaction may be:

BEGIN

SELECT FROM data table name WHERE attribute 1 is attribute value 1

SELECT FROM data table name WHERE attribute 2 is attribute value 2

END

Step 307, selecting a predetermined number of to-be-processed transactions from the at least one to-be-processed transaction as target to-be-processed transactions, and obtaining a target to-be-processed transaction set.

In some embodiments, the execution subject may select a predetermined number of pending transactions from the at least one pending transaction as target pending transactions, to obtain a target pending transaction set. When the predetermined number is the same as the number of the to-be-processed transactions in the at least one to-be-processed transaction, the execution main body may select a predetermined number of to-be-processed transactions from the at least one to-be-processed transaction as a target to-be-processed transaction at a preset time point, so as to obtain the target to-be-processed transaction set.

And 308, performing transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information and obtain a transaction conflict detection information set.

In some embodiments, the executing main body performs transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information, so as to obtain the transaction conflict detection information set, and may include the following steps:

the method comprises the following steps of firstly, hashing each record information in a record information group corresponding to each target to-be-processed transaction in a target to-be-processed transaction set through a conflict detection algorithm to generate a hash value to obtain a hash value group, and obtaining a hash value group set.

The collision detection algorithm may be an MD5(MD5 Message-Digest) algorithm. The collision detection algorithm may also be a CFD (conditional functional dependency) inconsistency detection algorithm.

As an example, the above-mentioned record information may be "0001, zhang san, man, 23 years old". The corresponding hash value may be "dd 7019a6950caa2c9ddb2dfb62e3a 445".

And secondly, in response to the fact that the two hash values with the same value exist in the hash value group set, generating transaction conflict detection information according to the transaction identification of the target to-be-processed transaction corresponding to each hash value in the two hash values, and obtaining the transaction conflict detection information set.

Wherein, the transaction conflict detection information may include: a first transaction identification, a second transaction identification, and a transaction conflict identification value. The first transaction identifier may be a transaction identifier corresponding to the first candidate pending transaction. The first candidate pending transaction may be a target pending transaction with a small timestamp in target pending transactions corresponding to each of the two hash values. The second transaction identifier may be a transaction identifier corresponding to a second candidate to-be-processed transaction. The second candidate pending transaction may be a target pending transaction with a large timestamp in the target pending transactions corresponding to each of the two hash values. The transaction conflict identification value may be used to characterize whether a transaction conflict exists.

As an example, the transaction conflict detection information may be [ a1, a2, 1 ]. Where "a 1" is the first transaction identification. "a 2" is the second transaction identification. "1" is a transaction conflict identification value. A "1" may indicate that a transaction conflict exists.

Step 309, generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set.

In some embodiments, the generating, by the execution agent, a directed acyclic graph based on the target pending transaction set and the transaction conflict detection information set may include:

and step one, generating a target graph based on the target to-be-processed transaction set.

The target graph may be a graph including only nodes and no edges. And the nodes in the target graph correspond to the target to-be-processed transactions in the target to-be-processed transaction set one by one.

As an example, the above target set of pending transactions may be [ a1, a2, a3, a4, a5 ]. The target map may be as shown in fig. 4. "a 1", "a 2", "a 3", "a 4", and "a 5" in fig. 4 denote respective nodes. Node "a 1" corresponds to target pending transaction "a 1". Node "a 2" corresponds to target pending transaction "a 2". Node "a 3" corresponds to target pending transaction "a 3". Node "a 4" corresponds to target pending transaction "a 4". Node "a 5" corresponds to target pending transaction "a 5".

And secondly, adding a directed edge in the target graph according to each transaction conflict detection information in the transaction conflict detection information set to generate the directed acyclic graph.

First, the execution subject may determine a section corresponding to the first transaction identifier included in the transaction conflict detection information as a source point. Then, the execution subject may determine, as the end point, a node corresponding to the second transaction identifier included in the transaction conflict detection information. Finally, the execution principal may add an oriented edge from the source point to the destination point to the target graph.

As an example, the transaction conflict detection information set may be { [ a1, a4, 1], [ a1, a2, 1], [ a2, a5, 1], [ a3, a4, 1] }. The directed acyclic graph may be as shown in fig. 5. In the above directed acyclic graph, a directed edge exists between the node "a 1" and the node "a 4". There is a directed edge between node "a 1" and node "a 2". There is a directed edge between node "a 2" and node "a 5". There is a directed edge between node "a 3" and node "a 4".

Step 310, performing topological sorting on the directed acyclic graph to generate a topological sorting sequence.

In some embodiments, the executing entity may topologically sort the directed acyclic graph through a topological sorting algorithm to generate the topological sort sequence.

The topological sorting algorithm may be a DFS (Depth-First-Search) algorithm. The topology ranking algorithm described above may also be the Kahn algorithm.

As an example, the topology order sequence described above may be as shown in fig. 6. Wherein, the topological ordered sequence in fig. 6 includes three groups of nodes [ (a1), (a2, a3), (a4, a5) ].

Step 311, executing the target to-be-processed transaction corresponding to each node in the topology sequencing sequence.

In some embodiments, the execution subject may execute the target pending transaction corresponding to each node in the topology ordering sequence.

As shown in fig. 6, the execution body may sequentially process target transactions to be processed corresponding to the "(a 1)" node, target transactions corresponding to the "(a 2, a 3)" two nodes, and target transactions corresponding to the "(a 4, a 5)" two nodes in a serial manner, in the [ (a1), (a2, a3), (a4, a5) ]. "(a 2, a 3)" the target transaction corresponding to the two nodes is executed concurrently. "(a 4, a 5)" the target transaction corresponding to the two nodes is executed concurrently.

As can be seen from fig. 3, compared with the description of some embodiments corresponding to fig. 2, first, when the concurrent execution type is the first concurrent execution type, each record in the data to be synchronized is synchronized to the backup database in a serial manner. Thereby ensuring that records can be written into the backup database on a per-entry basis. The situation that the data consistency is damaged possibly caused by concurrent backup is avoided. Secondly, when the concurrent execution type is a second concurrent execution type, converting each record of the data to be synchronized, including the full data, into a database insertion statement. In practice, the full amount of data refers to data that has been written to the data table. For such data, statements are inserted through the database and backed up to the backup database. For data to be synchronized, the data includes incremental data, and such data is often generated after data in a database is modified. Therefore, the corresponding candidate sub-database operation statements need to be generated according to the corresponding event types. In addition, concurrent synchronization often requires frequent read-write operations on the backup database, so as to avoid the read-write pressure of the backup database caused by frequent read-write operations. And merging the candidate sub-database operation statements meeting the merging condition to generate the sub-database operation statements. Therefore, the number of database operation statements is reduced, the read-write times of the backup database are reduced, and the read-write pressure of the backup database is reduced. In addition, when the concurrent execution type is a third concurrent execution type, performing transaction analysis on the data to be synchronized to generate at least one transaction to be processed. In practical situations, by generating the database operation statements and then executing the database operation statements, the execution period is relatively long, and the data synchronization efficiency is reduced. Therefore, at least one to-be-processed transaction is generated by performing transaction analysis on the to-be-synchronized data. However, transaction conflicts may exist between different pending transactions, which may cause database consistency to be broken. Therefore, through transaction conflict detection, the pending transaction with the transaction conflict is determined. And then, generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set. And further, carrying out topological sorting on the directed acyclic graph to generate a topological sorting sequence. And finally, executing the target to-be-processed transaction corresponding to each node in the topological sorting sequence. The data synchronization can be carried out in a serial mode by ensuring the target to-be-processed transaction with the transaction conflict, and the data synchronization can be carried out in a concurrent mode by ensuring the target to-be-processed transaction without the transaction conflict. By the method, the flexibility of data synchronization is greatly improved, the data synchronization requirements under different scenes are met, the data synchronization efficiency is improved, and the consumption of bandwidth resources is reduced.

With further reference to fig. 7, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a data synchronization apparatus, which correspond to those shown in fig. 2, and which may be applied in various electronic devices in particular.

As shown in fig. 7, a data synchronization apparatus 700 of some embodiments includes: a receiving unit 701, an acquiring unit 702, and a synchronizing unit 703. Wherein the receiving unit 701 is configured to receive the data synchronization task type information in response to a successful connection with the target database; an obtaining unit 702 configured to obtain data to be synchronized corresponding to the data synchronization task type information from the target database; a synchronizing unit 703 configured to synchronize the data to be synchronized to a backup database according to a concurrent execution type, where the backup database is used to backup data in the target database, and the concurrent execution type represents a manner of synchronizing the data to be synchronized to the backup database.

In some optional implementations of some embodiments, the obtaining unit 702 may be further configured to: and acquiring full data from the target database as the data to be synchronized in response to the fact that the data synchronization task type information is determined to be a first data synchronization type.

In some optional implementations of some embodiments, the obtaining unit 702 may be further configured to: and acquiring incremental data from the target database as data to be synchronized in response to the fact that the data synchronization task type information is determined to be a second data synchronization type.

In some optional implementations of some embodiments, the obtaining unit 702 may be further configured to: and acquiring full data and incremental data from the target database as the data to be synchronized in response to the determination that the data synchronization task type information is a third data synchronization type.

In some optional implementations of some embodiments, the synchronization unit 703 may be further configured to: and in response to determining that the concurrent execution type is a first concurrent execution type, synchronizing each record in the data to be synchronized to the backup database in a serial manner.

In some optional implementations of some embodiments, the synchronization unit 703 may be further configured to: in response to determining that the concurrent execution type is a second concurrent execution type, generating at least one database operation statement based on each record in the data to be synchronized; and executing each database operation statement in the at least one database operation statement.

In some optional implementations of some embodiments, the at least one database statement may include: at least one database insert statement; and the synchronization unit 703 may be further configured to: and in response to the fact that the data to be synchronized comprises full data, converting each record in the full data into a database insert statement to obtain at least one database insert statement.

In some optional implementations of some embodiments, the at least one database statement may further include: at least one sub-database operation statement; and the synchronization unit 703 may be further configured to: in response to the fact that the data to be synchronized comprises incremental data, generating candidate sub-database operation statements according to the event type corresponding to each record in the incremental data to obtain at least one candidate sub-database operation statement; and in response to determining that a target data table meets a first parallel processing condition, merging candidate sub-database operation statements meeting a merging condition in the at least one candidate sub-database operation statement to generate a sub-database operation statement, and obtaining the at least one sub-database operation statement, wherein the target data table is a data table corresponding to the data to be synchronized, the first parallel processing condition is that the target data table comprises a unique index or a main key, and the merging condition is that the main key or the unique index of the data tables operated by the candidate sub-database operation statements is the same.

In some optional implementations of some embodiments, the synchronization unit 703 may be further configured to: and executing each database operation statement in the at least one database operation statement in a serial working mode in response to determining that the target data table meets a second parallel processing condition, wherein the second parallel processing condition is that the target data table contains a unique index and a main key.

In some optional implementations of some embodiments, the synchronization unit 703 may be further configured to: in response to determining that the concurrent execution type is a third concurrent execution type, performing transaction analysis on the data to be synchronized to generate at least one transaction to be processed; determining a record information group corresponding to each transaction to be processed in the at least one transaction to be processed to obtain a record information group set; selecting a preset number of to-be-processed transactions from the at least one to-be-processed transaction as target to-be-processed transactions to obtain a target to-be-processed transaction set; performing transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information and obtain a transaction conflict detection information set; generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set, wherein nodes in the directed acyclic graph correspond to target to-be-processed transactions in the target to-be-processed transaction set one to one; carrying out topological sorting on the directed acyclic graph to generate a topological sorting sequence; and executing the target to-be-processed transaction corresponding to each node in the topology sequencing sequence.

Referring now to FIG. 8, a block diagram of an electronic device (such as computing device 101 shown in FIG. 1)800 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 8, an electronic device 800 may include a processing means (e.g., central processing unit, graphics processor, etc.) 801 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage means 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the electronic apparatus 800 are also stored. The processing apparatus 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

Generally, the following devices may be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; storage 808 including, for example, magnetic tape, hard disk, etc.; and a communication device 809. The communication means 809 may allow the electronic device 800 to communicate wirelessly or by wire with other devices to exchange data. While fig. 8 illustrates an electronic device 800 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 8 may represent one device or may represent multiple devices as desired.

In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through communications device 809, or installed from storage device 808, or installed from ROM 802. The computer program, when executed by the processing apparatus 801, performs the above-described functions defined in the methods of some embodiments of the present disclosure.

It should be noted that the computer readable medium described in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: receiving data synchronization task type information in response to successful connection with the target database; acquiring data to be synchronized corresponding to the data synchronization task type information from the target database; and synchronizing the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a mode of synchronizing the data to be synchronized to the backup database.

Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes a receiving unit, an acquiring unit, and a synchronizing unit. Where the names of these units do not in some cases constitute a limitation of the unit itself, for example, the receiving unit may also be described as a "unit that receives data synchronization task type information in case of successful connection with the target database".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims

1. A method of data synchronization, comprising:

receiving data synchronization task type information in response to successful connection with the target database;

acquiring data to be synchronized corresponding to the data synchronization task type information from the target database;

and synchronizing the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a mode of synchronizing the data to be synchronized to the backup database.

2. The method of claim 1, wherein the obtaining the data to be synchronized corresponding to the data synchronization task type information from the target database comprises:

and in response to the fact that the data synchronization task type information is determined to be a first data synchronization type, acquiring full data from the target database as the data to be synchronized.

3. The method of claim 1, wherein the obtaining the data to be synchronized corresponding to the data synchronization task type information from the target database comprises:

and acquiring incremental data from the target database as data to be synchronized in response to the fact that the data synchronization task type information is determined to be a second data synchronization type.

4. The method of claim 1, wherein the obtaining the data to be synchronized corresponding to the data synchronization task type information from the target database comprises:

and in response to the fact that the data synchronization task type information is determined to be a third data synchronization type, acquiring full data and incremental data from the target database as the data to be synchronized.

5. The method of claim 1, wherein the synchronizing the data to be synchronized to a backup database according to a concurrent execution type comprises:

and in response to determining that the concurrent execution type is a first concurrent execution type, synchronizing each record in the data to be synchronized to the backup database in a serial manner.

6. The method according to one of claims 2 to 4, wherein the synchronizing the data to be synchronized to a backup database according to a concurrent execution type comprises:

in response to determining that the concurrent execution type is a second concurrent execution type, generating at least one database operation statement based on each record in the data to be synchronized; and executing each database operation statement in the at least one database operation statement.

7. The method of claim 6, wherein the at least one database statement comprises: at least one database insert statement; and

generating at least one database operation statement based on each record in the data to be synchronized, including:

and in response to the fact that the data to be synchronized comprises full data, converting each record in the full data into a database insertion statement to obtain at least one database insertion statement.

8. The method of claim 7, wherein the at least one database statement further comprises: at least one sub-database operation statement; and

generating at least one database operation statement based on each record in the data to be synchronized, further comprising:

in response to the fact that the data to be synchronized comprises incremental data, generating candidate sub-database operation statements according to the event type corresponding to each record in the incremental data to obtain at least one candidate sub-database operation statement;

in response to determining that a target data table meets a first parallel processing condition, merging candidate sub-database operation statements meeting a merging condition in the at least one candidate sub-database operation statement to generate a sub-database operation statement, and obtaining the at least one sub-database operation statement, wherein the target data table is a data table corresponding to the data to be synchronized, the first parallel processing condition is that the target data table includes a unique index or a main key, and the merging condition is that the main key or the unique index of the data tables operated by the candidate sub-database operation statements are the same.

9. The method of claim 8, wherein said executing each of the at least one database operation statement comprises:

executing each of the at least one database operation statement in a serial working manner in response to determining that a target data table satisfies a second parallel processing condition, wherein the second parallel processing condition is that the target data table contains a unique index and contains a primary key.

10. The method according to one of claims 2 to 4, wherein the synchronizing the data to be synchronized to a backup database according to a concurrent execution type comprises:

in response to determining that the concurrent execution type is a third concurrent execution type, performing transaction analysis on data to be synchronized to generate at least one transaction to be processed;

determining a record information group corresponding to each to-be-processed transaction in the at least one to-be-processed transaction to obtain a record information group set;

selecting a preset number of to-be-processed transactions from the at least one to-be-processed transaction as target to-be-processed transactions to obtain a target to-be-processed transaction set;

performing transaction conflict detection on the target to-be-processed transaction according to the record information group corresponding to each target to-be-processed transaction in the target to-be-processed transaction set to generate transaction conflict detection information and obtain a transaction conflict detection information set;

generating a directed acyclic graph based on the target to-be-processed transaction set and the transaction conflict detection information set, wherein nodes in the directed acyclic graph correspond to target to-be-processed transactions in the target to-be-processed transaction set one to one;

performing topological sorting on the directed acyclic graph to generate a topological sorting sequence;

and executing the target to-be-processed transaction corresponding to each node in the topological sorting sequence.

11. A data synchronization apparatus, comprising:

a receiving unit configured to receive data synchronization task type information in response to a successful connection with a target database;

an obtaining unit configured to obtain data to be synchronized corresponding to the data synchronization task type information from the target database;

the synchronization unit is configured to synchronize the data to be synchronized to a backup database according to a concurrent execution type, wherein the backup database is used for backing up the data in the target database, and the concurrent execution type represents a manner of synchronizing the data to be synchronized to the backup database.

12. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-10.

13. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1 to 10.