CN114281892A

CN114281892A - Data synchronization method, device, equipment and storage medium

Info

Publication number: CN114281892A
Application number: CN202111572700.4A
Authority: CN
Inventors: 姚乔兵; 徐翔; 冯哲
Original assignee: China Unionpay Co Ltd
Current assignee: China Unionpay Co Ltd
Priority date: 2021-12-21
Filing date: 2021-12-21
Publication date: 2022-04-05

Abstract

The embodiment of the application provides a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium, and relates to the technical field of cloud computing, wherein the method comprises the following steps: the data synchronization system acquires a plurality of data packets to be synchronized in the target time window from the source database, so that the data packets to be synchronized in the source database can be synchronized in time. The data synchronization system determines the target data records corresponding to the multiple data packets to be synchronized in parallel and determines the data identification of each target data record. And for each data packet to be synchronized, determining a target database corresponding to each of the plurality of data records based on the data identifiers of the plurality of target data records corresponding to the data packet to be synchronized, and respectively storing the plurality of data records into the corresponding target database. The data synchronization system processes a plurality of data packets to be synchronized in parallel, improves the read-write throughput during data synchronization, and reduces the data synchronization delay.

Description

Data synchronization method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of cloud computing, in particular to a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium.

Background

Different online transaction data exist in different bank data systems, and in order to realize payment clearing among different banks, the online transaction data in the different bank data systems need to be transferred to a cross-bank clearing system. Because daily online transaction data is more than 3.5 hundred million, a safe, reliable and excellent-performance data synchronization scheme is required to be established between the cross-bank clearing system and different bank data systems.

In the related technology, the data transfer component is adopted to transfer the online transaction data in the bank data system to the cross-bank clearing system, but when the data transfer component transfers the online transaction data, the read-write throughput supported by a single database is limited, and the data synchronization delay is very high.

Disclosure of Invention

The embodiment of the application provides a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium, which are used for solving the problem of data synchronization delay.

In one aspect, an embodiment of the present application provides a data synchronization method, where the method includes:

acquiring a plurality of data packets to be synchronized in a target time window from a source database;

determining target data records corresponding to the multiple data packets to be synchronized in parallel, and determining a data identifier of each target data record;

and for each data packet to be synchronized, determining a target database corresponding to each of a plurality of data records based on the data identifiers of the plurality of target data records corresponding to the data packet to be synchronized, and respectively storing the plurality of data records into the corresponding target database.

Optionally, the determining, in parallel, target data records corresponding to the multiple data packets to be synchronized, and determining a data identifier of each target data record includes:

determining the plurality of data packets to be synchronized in parallel for splitting through a plurality of processes to obtain original data records corresponding to the plurality of data packets to be synchronized;

and performing deformation processing on each obtained original data record to obtain target data records corresponding to the multiple data packets to be synchronized, and determining a data identifier of each target data record, wherein the multiple processes are processes in one or more application nodes.

Optionally, the determining, based on the data identifiers of the multiple data records corresponding to the data packet to be synchronized, the target databases corresponding to the multiple data records, and storing the multiple data records into the corresponding target databases respectively includes:

and searching a routing table by adopting the data identifiers of the target data records corresponding to the data packet to be synchronized through a process, determining the target databases corresponding to the target data records respectively, and storing the target data records into the corresponding target databases respectively.

the data packets to be synchronized are split in parallel through a plurality of processes and sub-processes corresponding to the processes, and original data records corresponding to the data packets to be synchronized are obtained;

Optionally, the splitting the multiple data packets to be synchronized in parallel through multiple processes and multiple sub-processes corresponding to the multiple processes to obtain the original data records corresponding to the multiple data packets to be synchronized includes:

for the plurality of processes, performing in parallel the steps of:

acquiring at least one target synchronization data packet from the plurality of data packets to be synchronized through a process;

and splitting the at least one target synchronous data packet in parallel through a plurality of primary sub-threads corresponding to the process to obtain a plurality of original data records corresponding to the at least one target synchronous data packet.

Optionally, after acquiring, by a process, at least one target synchronization packet from the multiple packets to be synchronized, the method further includes:

storing the at least one target isochronous packet in a packet block queue;

before the splitting the at least one target synchronization data packet in parallel through the plurality of primary sub-threads corresponding to the process to obtain the plurality of original data records corresponding to the at least one target synchronization data packet, the method further includes:

and acquiring the at least one target synchronous data packet from the data packet blocking queue through the plurality of primary sub-threads.

Optionally, the performing deformation processing on each obtained original data record to obtain target data records corresponding to the multiple data packets to be synchronized, and determining a data identifier of each target data record includes:

and performing deformation processing on each obtained original data record in parallel through a plurality of secondary sub-threads corresponding to the process to obtain target data records corresponding to the plurality of data packets to be synchronized, and determining data identifications of each obtained target data record.

Optionally, after the splitting the at least one target synchronization data packet in parallel through the plurality of primary sub-threads corresponding to the process to obtain a plurality of original data records corresponding to the at least one target synchronization data packet, the method further includes:

respectively adding the obtained original data records to a data record blocking queue through the plurality of primary sub threads;

before performing deformation processing on each obtained original data record in parallel through a plurality of secondary sub-threads corresponding to the process to obtain a target data record corresponding to each of the plurality of data packets to be synchronized, the method further includes:

and acquiring original data records from the data record blocking queue through the plurality of secondary sub-threads.

Optionally, the determining, based on the data identifiers of the multiple target data records corresponding to the data packet to be synchronized, the target databases corresponding to the multiple data records, and storing the multiple data records into the corresponding target databases respectively includes:

and searching a routing table by using the data identifications of the target data records corresponding to the data packet to be synchronized through a plurality of secondary sub-threads, determining the target database corresponding to each target data record, and respectively storing the target data records into the corresponding target database.

Optionally, the storing the plurality of target data records into corresponding target databases respectively includes:

for the plurality of target data records, respectively performing the following steps:

acquiring connection information of a target database corresponding to a target data record from a database connection pool;

and storing the target data record into a corresponding target database based on the connection information.

In one aspect, an embodiment of the present application provides a data synchronization apparatus, where the apparatus includes:

the acquisition module is used for acquiring a plurality of data packets to be synchronized in a target time window from a source database;

the determining module is used for determining target data records corresponding to the multiple data packets to be synchronized in parallel and determining a data identifier of each target data record;

and the storage module is used for determining a target database corresponding to each data record based on the data identification of the target data records corresponding to each data packet to be synchronized, and storing the data records into the corresponding target databases respectively.

Optionally, the determining module is specifically configured to:

Optionally, the saving module is specifically configured to:

Optionally, the determining module is specifically configured to:

for the plurality of processes, performing in parallel the steps of:

Optionally, the system further comprises an adding module, and the adding module is specifically configured to:

after at least one target synchronous data packet is obtained from the plurality of data packets to be synchronized through one process, the at least one target synchronous data packet is stored in a data packet blocking queue;

the acquisition module is further configured to:

and acquiring the at least one target synchronous data packet from a data packet blocking queue through the plurality of primary sub-threads before the at least one target synchronous data packet is split in parallel through the plurality of primary sub-threads corresponding to the process to obtain a plurality of original data records corresponding to the at least one target synchronous data packet.

Optionally, the determining module is specifically configured to:

Optionally, the adding module is further configured to:

after the at least one target synchronous data packet is split in parallel through a plurality of primary sub-threads corresponding to the process to obtain a plurality of original data records corresponding to the at least one target synchronous data packet, the obtained original data records are added to a data record blocking queue through the plurality of primary sub-threads;

the acquisition module is further configured to:

and acquiring original data records from the data record blocking queue through the plurality of secondary sub-threads before performing deformation processing on each acquired original data record in parallel through the plurality of secondary sub-threads corresponding to the process to acquire target data records corresponding to the plurality of data packets to be synchronized.

Optionally, the saving module is specifically configured to:

In one aspect, an embodiment of the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the data synchronization method when executing the program.

In one aspect, embodiments of the present application provide a computer-readable storage medium, which stores a computer program executable by a computer device, and when the program runs on the computer device, the computer device is caused to execute the steps of the data synchronization method.

In the embodiment of the application, the data synchronization system obtains a plurality of data packets to be synchronized in the target time window from the source database, so that the data packets to be synchronized in the source database can be synchronized in time. The data synchronization system determines the target data records corresponding to the multiple data packets to be synchronized in parallel and determines the data identification of each target data record. And for each data packet to be synchronized, determining a target database corresponding to each of the plurality of data records based on the data identifiers of the plurality of target data records corresponding to the data packet to be synchronized, and respectively storing the plurality of data records into the corresponding target database. The data synchronization system processes a plurality of data packets to be synchronized in parallel, improves the read-write throughput during data synchronization, and reduces the data synchronization delay.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic diagram of a system architecture according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a data synchronization method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a data synchronization method according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of an application center provided in an embodiment of the present application;

fig. 5 is a schematic flowchart of a data synchronization method according to an embodiment of the present application;

fig. 6 is a schematic flowchart of a data synchronization method according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a data synchronization system according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more clearly apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, a system architecture diagram applicable to the embodiment of the present application is shown, where the system architecture includes at least a terminal device 101, a data synchronization system 102, a source database 103, and a target database 104.

The terminal apparatus 101 is installed with a target application for data synchronization, which may be a client installed in advance, a web page version application, or an applet embedded in other applications, or the like. The terminal device 101 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like.

The data synchronization system 102, the source database 103 and the target database 104 are background servers of the target application and provide services for the target application. The data synchronization system 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, Content Delivery Network (CDN), big data and an artificial intelligence platform.

The source database 103 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.

The target database 104 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, Content Delivery Network (CDN), big data and artificial intelligence platform, and the like.

The terminal device 101 and the data synchronization system 102 may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein. The data synchronization system 102 and the source database 103 may be directly or indirectly connected through wired or wireless communication, and the application is not limited herein. The data synchronization system 102 and the target database 104 may be directly or indirectly connected through wired or wireless communication, and the application is not limited thereto.

The terminal device 101 responds to the data synchronization operation of the user and sends a data synchronization instruction to the data synchronization system 102. The data synchronization system 102 receives the data synchronization command, and obtains a plurality of data packets to be synchronized within the target time window from the source database 103. The data synchronization system 102 determines, in parallel, target data records corresponding to a plurality of data packets to be synchronized, and determines a data identifier of each target data record. For each data packet to be synchronized, the target database 104 corresponding to each of the plurality of data records is determined based on the data identifiers of the plurality of target data records corresponding to the data packet to be synchronized, and the plurality of data records are respectively stored in the corresponding target database 104.

Based on the system architecture diagram shown in fig. 1, an embodiment of the present application provides a flow of a data synchronization method, as shown in fig. 2, where the flow of the method is executed by the data synchronization system 102 shown in fig. 1, and includes the following steps:

step S201, obtaining a plurality of data packets to be synchronized in a target time window from a source database.

Specifically, the starting time point of the target time window is related to the time point when the data packet to be synchronized is saved in the source database. If a plurality of data packets to be synchronized are stored in the source database at 7:00:00, the target time window is 7:00:00-7:00:01 (including the time point 7:00:00, excluding the time point 7:00: 01). And acquiring a plurality of data packets to be synchronized in a target time window 7:00:00-7:00:01 (including a time point 7:00:00 and not including the time point 7:00:01) from a source database.

After acquiring a plurality of data packets to be synchronized in the target time window from the source database, the target time window needs to be determined again. And the newly determined starting time point of the target time window is the ending time point of the original target time window. The sizes of the different target time windows may be the same or different.

For example, the first target time window is 10:00:00-10:00:01 (including time point 10:00:00, excluding time point 10:00: 01). After obtaining a plurality of data packets to be synchronized in the first target time window from the source database, determining that the second target time window is 10:00:01-10:00:02 (including time point 10:00:01, excluding time point 10:00:02), obtaining a plurality of data packets to be synchronized in the second target time window from the source database, and so on.

Step S202, determining target data records corresponding to a plurality of data packets to be synchronized in parallel, and determining a data identifier of each target data record.

Specifically, each packet to be synchronized corresponds to at least one target data record. Each target data record corresponds to a data identifier.

Part of identification fields in the target data records can be selected as data identifications; part of identification fields in the target data records can be selected for carrying out Hash coding to obtain coding fields, and the coding fields are used as data identifications; the hash coding can be directly carried out on the target data record to obtain the coding field, and the coding field is used as the data identification.

For example, as shown in table 1, the target data record selects the order number 001 as the identification field to perform hash coding to obtain the coded field 0xabc, and uses the coded field 0xabc as the data identification.

Table 1.

Order numbering	Name of commodity	Credit card number	Name of card holder
				001	A	C1122	Zhang San

Step S203, for each data packet to be synchronized, determining a target database corresponding to each of the plurality of data records based on the data identifiers of the plurality of target data records corresponding to the data packet to be synchronized, and storing the plurality of data records into the corresponding target databases respectively.

For example, as shown in fig. 3, the data synchronization system processes 2 data packets to be synchronized in parallel, namely, data packet 1 to be synchronized and data packet 2 to be synchronized. And determining 2 entry mark data records corresponding to the data packet 1 to be synchronized, namely a target data record 1 and a target data record 2. The data of target data record 1 is identified as A and the data of target data record 2 is identified as B. And based on the data identification A, determining a target database corresponding to the target data record 1 as a database A, and storing the target data record 1 to the database A. And based on the data identifier B, determining that the target database corresponding to the target data record 2 is the database B, and storing the target data record 2 to the database A.

The data packet 2 to be synchronized corresponds to the item number data record 1, i.e. the target data record 3, and the data identifier of the target data record 3 is a. And based on the data identification A, determining a target database corresponding to the target data record 3 as a database A, and storing the target data record 3 in the database A.

The embodiment of the application provides two schemes for realizing parallel processing of a plurality of data packets to be synchronized:

in one possible embodiment, parallel processing of a plurality of data packets to be synchronized is implemented by a plurality of processes.

Optionally, in step S202, a plurality of data packets to be synchronized are determined in parallel through a plurality of processes to be split, so as to obtain original data records corresponding to the plurality of data packets to be synchronized; and carrying out deformation processing on each obtained original data record, obtaining target data records corresponding to a plurality of data packets to be synchronized respectively, and determining the data identifier of each target data record.

Specifically, the deformation processing of the original data record includes modification, filling and the like of partial fields in the original data record.

The plurality of processes are processes in one or more application nodes. The plurality of application nodes may be in one application center or in a plurality of application centers. An application center includes a plurality of application nodes and a plurality of databases. One application node corresponds to a plurality of databases. One application node can simultaneously start a plurality of processes.

For example, as shown in fig. 4, the two application centers are application center 1 and application center 2, respectively. The application center 1 comprises 2 application nodes and 3 databases, wherein the 2 application nodes are respectively an application node 1 and an application node 2, and the 3 databases are respectively a database 1, a database 2 and a database 3. The application node 1 corresponds to the database 1, the database 2, and the database 3. The application node 2 corresponds to the database 1, the database 2, and the database 3.

The application center 2 comprises 1 application node and 1 database, wherein 1 application node is an application node 3, and 1 database is a database 4. The application node 3 corresponds to the database 4.

In the embodiment of the present application, since there are a plurality of application centers, each application center includes a plurality of application nodes. When all the application nodes in one application center are down or one application node in one application center is down, the data synchronization system is not influenced to carry out data synchronization work, and the disaster recovery capability of the data synchronization system is enhanced. Meanwhile, the application center and the application nodes can be adaptively increased according to the condition of the service data, so that the universality of the data synchronization system is enhanced. The plurality of application centers and the plurality of application nodes in each application center can effectively share the service data volume, realize load balance, improve the read-write throughput during data synchronization and reduce the data synchronization delay.

Optionally, in step S203, a process is performed to search the routing table by using the data identifiers of the multiple target data records corresponding to the data packet to be synchronized, determine a target database corresponding to each of the multiple target data records, and store the multiple target data records into the corresponding target databases respectively.

Specifically, the routing table stores the corresponding relationship between each data identifier and the target database, and may be one routing table maintained by each process, or one routing table maintained by each application node.

For a plurality of target data records, respectively performing the following steps:

and acquiring the connection information of the target database corresponding to one target data record from the database connection pool, and storing one target data record into the corresponding target database based on the connection information.

In the embodiment of the application, each process in the data synchronization system is not connected with each target database in advance, but is connected with the target database after determining the target database corresponding to the target data record, so that the process concurrence number in each application node is decoupled from the target database, and the universality of the data synchronization system is enhanced.

For example, as shown in fig. 5, 2 processes are started on the application node 1 in the application center 1, and 2 data packets to be synchronized are processed in parallel through the process 1 and the process 2. The data packet 1 to be synchronized is split through the process 1, and 2 original data records, namely the original data record 1 and the original data record 2, are obtained. The original data record 1 is subjected to deformation processing through the process 1 to obtain a target data record 1, and the data identifier of the target data record 1 is determined to be A. And performing deformation processing on the original data record 2 through the process 1 to obtain a target data record 2, and determining that the data identifier of the target data record 2 is B.

The data packet 2 to be synchronized is split through the process 2, and 1 original data record, namely the original data record 3, is obtained. The original data record 3 is deformed by the process 2 to obtain the target data record 3, and the data identifier of the target data record 3 is determined to be a.

The routing table maintained by the application node 1 is shown in table 2, the data identifier of the target data record 1 is a, and the target database corresponding to the target data record 1 is determined as database a by searching the routing table. And acquiring the connection information of the database A from the database connection pool, and storing the target data record 1 into the database A based on the connection information.

The data identifier of the target data record 2 is B, and the target database corresponding to the target data record 2 is determined as database B by searching the routing table. And acquiring the connection information of the database B from the database connection pool, and storing the target data record 2 into the database B based on the connection information.

The data identifier of the target data record 3 is a, and the target database corresponding to the target data record 3 is determined as a database a by searching the routing table. And acquiring the connection information of the database A from the database connection pool, and storing the target data record 3 into the database A based on the connection information.

Table 2.

Data identification	Object database
		A	Database A
B	Database B
		C	Database C

In the embodiment of the application, a plurality of processes process a plurality of data packets to be synchronized in parallel, so that the processing speed of the data packets to be synchronized is improved, and the read-write throughput of the data synchronization system is enhanced.

According to a possible implementation mode, parallel processing of a plurality of data packets to be synchronized is achieved through a plurality of processes and sub-threads corresponding to the processes.

Optionally, in step S202, determining, in parallel, target data records corresponding to a plurality of packets to be synchronized, and determining a data identifier of each target data record, includes the following steps:

step S301, a plurality of data packets to be synchronized are split in parallel through a plurality of processes and sub-processes corresponding to the plurality of processes, and original data records corresponding to the plurality of data packets to be synchronized are obtained.

Specifically, for a plurality of processes, the following steps are performed in parallel:

and acquiring at least one target synchronous data packet from a plurality of data packets to be synchronized through a process, and storing the at least one target synchronous data packet into a data packet blocking queue.

And acquiring at least one target synchronous data packet from the data packet blocking queue through a plurality of primary sub-threads corresponding to the process, and splitting the at least one target synchronous data packet in parallel to acquire a plurality of original data records corresponding to the at least one target synchronous data packet.

And finally, adding the obtained original data records into a data record blocking queue through a plurality of primary sub-threads.

Step S302, performing deformation processing on each obtained original data record, obtaining target data records corresponding to a plurality of data packets to be synchronized, and determining a data identifier of each target data record.

Wherein the plurality of processes are processes in one or more application nodes. The plurality of application nodes may be in one application center or in a plurality of application centers. An application center includes a plurality of application nodes and a plurality of databases. One application node corresponds to a plurality of databases. One application node can simultaneously start a plurality of processes.

and acquiring original data records from the data record blocking queue through a plurality of secondary sub-threads corresponding to one process.

And performing deformation processing on each obtained original data record in parallel through a plurality of secondary sub-threads to obtain target data records corresponding to a plurality of data packets to be synchronized and determine data identifications of each obtained target data record.

Optionally, in step S203, a routing table is searched by using the data identifiers of the multiple target data records corresponding to the data packet to be synchronized through the multiple secondary child threads, a target database corresponding to each of the multiple target data records is determined, and the multiple target data records are respectively stored in the corresponding target databases.

Specifically, the routing table stores the corresponding relationship between each data identifier and the target database, and may be one routing table maintained by each process, or one routing table maintained by each application node. Because the routing table is maintained by each process or each application node, the routing table cannot be modified by a plurality of secondary sub-threads corresponding to each process, and the problem that other secondary sub-threads cannot query the routing table because one secondary sub-thread modifies the routing table is avoided.

In the embodiment of the application, each secondary sub-thread in the data synchronization system is not connected with each target database in advance, but is connected with the target database after determining the target database corresponding to the target data record, so that the process concurrency number in each application node is decoupled from the target database, and the universality of the data synchronization system is enhanced.

For example, as shown in fig. 6, 2 processes are started on the application node 1 in the application center 1, and the process 1 and the process 2 process 4 data packets to be synchronized in parallel. Through the process 1, 2 target synchronous data packets are acquired from 4 data packets to be synchronized, namely the target synchronous data packet 1 and the target synchronous data packet 2, and the 2 target synchronous data packets are stored in a data packet blocking queue corresponding to the process 1. Through the process 2, 2 target synchronous data packets, namely a target synchronous data packet 3 and a target synchronous data packet 4, are obtained from 4 data packets to be synchronized, and the 2 target synchronous data packets are stored in a data packet blocking queue corresponding to the process 2.

And aiming at the data packet blocking queue corresponding to the process 1, acquiring 2 target synchronous data packets from the data packet blocking queue through a plurality of primary sub-threads corresponding to the process 1, and processing the target data packets in parallel. Namely, the target synchronous data packet 1 is split through the primary sub-thread 1, the original data record 1 and the original data record 2 corresponding to the target synchronous data packet 1 are obtained, and the original data record 1 and the original data record 2 are respectively added to the data record blocking queue. Meanwhile, the target synchronous data packet 2 is split through the primary sub-thread 2, the original data record 3 corresponding to the target synchronous data packet 2 is obtained, and the original data record 3 is added to the data record blocking queue.

Through a plurality of secondary sub-threads corresponding to the process 1, the secondary sub-thread 1 obtains an original data record 1 from the data record blocking queue, the secondary sub-thread 2 obtains an original data record 2 from the data record blocking queue, and the secondary sub-thread 3 obtains an original data record 3 from the data record blocking queue. And the secondary sub thread 1, the secondary sub thread 2 and the secondary sub thread 3 process each original data record in parallel. That is, the secondary sub-thread 1 performs deformation processing on the original data record 1 to obtain the target data record 1, and determines that the data identifier of the target data record 1 is a. And the secondary sub-thread 2 carries out deformation processing on the original data record 2 to obtain a target data record 2, and the data identifier of the target data record 2 is determined to be B. The secondary sub-thread 3 performs deformation processing on the original data record 3 to obtain a target data record 3, and determines that the data identifier of the target data record 3 is A.

The routing table maintained by the process 1 is shown in table 2, the data identifier of the target data record 1 is a, and the target database corresponding to the target data record 1 is determined as the database a by searching the routing table. And acquiring the connection information of the database A from the database connection pool, and storing the target data record 1 to the database A based on the connection information.

The data identifier of the target data record 2 is B, and the target database corresponding to the target data record 2 is determined as database B by searching the routing table. And acquiring the connection information of the database B from the database connection pool, and storing the target data record 2 to the database B based on the connection information.

The data identifier of the target data record 3 is a, and the target database corresponding to the target data record 3 is determined as a database a by searching the routing table. And acquiring the connection information of the database A from the database connection pool, and storing the target data record 3 to the database A based on the connection information.

For the data packet blocking queue corresponding to the process 2, the processing procedure is the same as that of the data packet blocking queue corresponding to the process 1, and details are not described here.

In the embodiment of the application, the data packets to be synchronized are processed in parallel by a plurality of processes, so that the processing speed of the data packets is improved, and the data synchronization throughput of the data synchronization system is enhanced.

And splitting the target synchronous data packets through the primary sub-thread to obtain a plurality of original data records corresponding to each target synchronous data packet. And processing each original data record through a secondary sub-thread to obtain a target data record, and adding each target data record into a corresponding target database. And a two-stage sub-thread concurrent processing mode is adopted, so that the processing time consumption of a single target synchronous data packet is further improved, and the system throughput of data synchronization is effectively improved.

For better explaining the embodiments of the present application, a data synchronization method provided by the embodiments of the present application is described below with reference to specific implementation scenarios, where the method is performed by the data synchronization system 102 in fig. 1. As shown in fig. 7, the system architecture diagram of the data synchronization system 102 includes a PUB end 701, a server end 702, and a SUB end 703.

The PUB 701 acquires a plurality of data packets to be synchronized located in a target time window from a source database, and sends the plurality of data packets to be synchronized to the server 702.

The server 702 receives and stores a plurality of data packets to be synchronized, and simultaneously sends the plurality of data packets to be synchronized to the SUB 703.

The SUB-end 703 includes a plurality of application centers, each including a plurality of application nodes and corresponding databases. Each application node starts multiple processes. For a process in one of the application nodes, at least one target synchronization data packet is obtained from a plurality of data packets to be synchronized sent by the server 702 through the process, and the at least one target synchronization data packet is stored in a data packet blocking queue corresponding to the process. And acquiring at least one target synchronous data packet from the data packet blocking queue through a plurality of primary sub-threads corresponding to the process, and splitting the at least one target synchronous data packet in parallel to acquire a plurality of original data records corresponding to the at least one target synchronous data packet. And adding the obtained original data records to a data record blocking queue through a plurality of primary sub threads respectively.

And acquiring original data records from the data record blocking queue through a plurality of secondary sub-threads corresponding to one process. And performing deformation processing on each obtained original data record in parallel through a plurality of secondary sub-threads to obtain target data records corresponding to a plurality of data packets to be synchronized and determine data identifications of each obtained target data record.

And finally, searching a routing table by using the data identifications of the target data records corresponding to the data packet to be synchronized through a plurality of secondary sub-threads, determining a target database corresponding to each target data record, and respectively storing the target data records into the corresponding target database. The secondary sub-thread sends the processing result of each target data record to each primary sub-thread, each primary sub-thread determines the processing result of each target synchronous data packet according to the processing result of each target data record, and each primary sub-thread sends the processing result of each target synchronous data packet to the server 702.

If the target data record is not stored through one secondary sub-thread, determining a target synchronous data packet where the target data record is located, and sending a processing failure result of the target data record to a primary sub-thread which processes the target synchronous data packet. Determining that the processing result of the target synchronization packet is a failure through the primary child thread, sending the processing failure result of the target synchronization packet to the server 702, and sending the target synchronization packet to the SUB terminal 703 again by the server 702.

And if the target data record is successfully stored through one secondary sub-thread, determining a target synchronous data packet where the target data record is located, and sending a processing success result of the target data record to a primary sub-thread for processing the target synchronous data packet. And when the processing results of all the target synchronous data packets split from the target synchronous data packet are determined to be successful through the primary sub-thread, the successful processing results of the target synchronous data packets are sent to the server 702.

In the embodiment of the application, the data synchronization system obtains a plurality of data packets to be synchronized in the target time window from the source database, so that the data packets to be synchronized in the source database can be synchronized in time.

The data packets to be synchronized are processed in parallel by a plurality of processes, so that the processing speed of the data packets is improved, and the data synchronization throughput of the data synchronization system is enhanced. And splitting the target synchronous data packets through the primary sub-thread to obtain a plurality of original data records corresponding to each target synchronous data packet. And processing each original data record through a secondary sub-thread to obtain a target data record, and adding each target data record into a corresponding target database. And a two-stage sub-thread concurrent processing mode is adopted, so that the processing time consumption of a single target synchronous data packet is further improved, and the system throughput of data synchronization is effectively improved.

Each secondary sub-thread is not connected with each target database in advance, decoupling of the process concurrency number in each application node and the target database is achieved, and universality of the data synchronization system is enhanced.

Based on the same technical concept, an embodiment of the present application provides a data synchronization apparatus, as shown in fig. 8, the data synchronization apparatus 800 includes:

an obtaining module 801, configured to obtain multiple data packets to be synchronized in a target time window from a source database;

a determining module 802, configured to determine, in parallel, target data records corresponding to the multiple data packets to be synchronized, and determine a data identifier of each target data record;

a saving module 803, configured to determine, for each to-be-synchronized data packet, a target database corresponding to each of the multiple data records based on the data identifiers of the multiple target data records corresponding to the to-be-synchronized data packet, and respectively save the multiple data records in the corresponding target databases.

Optionally, the determining module 802 is specifically configured to:

Optionally, the saving module 803 is specifically configured to:

Optionally, the determining module 802 is specifically configured to:

for the plurality of processes, performing in parallel the steps of:

Optionally, the system further includes an adding module 804, where the adding module 804 is specifically configured to:

the obtaining module 801 is further configured to:

Optionally, the determining module 802 is specifically configured to:

Optionally, the adding module 804 is further configured to:

the obtaining module 801 is further configured to:

Optionally, the saving module 803 is specifically configured to:

Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, as shown in fig. 9, and includes at least one processor 901 and a memory 902 connected to the at least one processor, where a specific connection medium between the processor 901 and the memory 902 is not limited in the embodiment of the present application, and the processor 901 and the memory 902 are connected through a bus in fig. 9 as an example. The bus may be divided into an address bus, a data bus, a control bus, etc.

In the embodiment of the present application, the memory 902 stores instructions executable by the at least one processor 901, and the at least one processor 901 may execute the steps included in the data synchronization method by executing the instructions stored in the memory 902.

The processor 901 is a control center of the computer device, and can connect various parts of the computer device by using various interfaces and lines, and perform data synchronization by executing or executing instructions stored in the memory 902 and calling data stored in the memory 902. Optionally, the processor 901 may include one or more processing units, and the processor 901 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application program, and the like, and the modem processor mainly processes wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 may be implemented on the same chip, or in some embodiments, they may be implemented separately on separate chips.

The processor 901 may be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an Application Specific Integrated Circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, and may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present Application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in a processor.

Memory 902, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory 902 may include at least one type of storage medium, and may include, for example, a flash Memory, a hard disk, a multimedia card, a card-type Memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a charge Erasable Programmable Read Only Memory (EEPROM), a magnetic Memory, a magnetic disk, an optical disk, and so on. The memory 902 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 902 of the embodiments of the present application may also be circuitry or any other device capable of performing a storage function for storing program instructions and/or data.

Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when the program is run on the computer device, causes the computer device to perform the steps of the above-mentioned data synchronization method.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method of data synchronization, comprising:

2. The method of claim 1, wherein the determining, in parallel, target data records corresponding to the plurality of packets to be synchronized, and determining a data identifier of each target data record, comprises:

3. The method according to claim 2, wherein the determining a target database corresponding to each of the plurality of data records based on the data identifiers of the plurality of data records corresponding to the data packet to be synchronized, and storing the plurality of data records into the corresponding target databases respectively comprises:

4. The method of claim 1, wherein the determining, in parallel, target data records corresponding to the plurality of packets to be synchronized, and determining a data identifier of each target data record, comprises:

5. The method according to claim 4, wherein the obtaining of the original data records corresponding to the multiple data packets to be synchronized by splitting the multiple data packets to be synchronized in parallel through multiple processes and corresponding sub-threads of the multiple processes comprises:

for the plurality of processes, performing in parallel the steps of:

6. The method as claimed in claim 5, wherein said obtaining, by a process, at least one target synchronization packet from said plurality of packets to be synchronized further comprises:

storing the at least one target isochronous packet in a packet block queue;

7. The method according to claim 5, wherein the performing deformation processing on the obtained original data records to obtain target data records corresponding to the multiple data packets to be synchronized, and determining the data identifier of each target data record includes:

8. The method according to claim 7, wherein the splitting the at least one target synchronization packet in parallel through a plurality of primary child threads corresponding to the one process to obtain a plurality of original data records corresponding to each of the at least one target synchronization packet, further comprises:

9. The method as claimed in claim 7, wherein the determining a target database corresponding to each of the plurality of data records based on the data identifier of the plurality of target data records corresponding to the data packet to be synchronized, and storing the plurality of data records into the corresponding target database respectively comprises:

10. The method of any one of claims 1 to 9, wherein said storing said plurality of target data records into respective target databases comprises:

11. A data synchronization apparatus, comprising:

12. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method of any of claims 1 to 10 are performed when the program is executed by the processor.

13. A computer-readable storage medium, having stored thereon a computer program executable by a computer device, for causing the computer device to perform the steps of the method of any one of claims 1 to 10, when the program is run on the computer device.