CN112035571A - Data synchronization method, device, equipment and storage medium - Google Patents
Data synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112035571A CN112035571A CN202010838819.0A CN202010838819A CN112035571A CN 112035571 A CN112035571 A CN 112035571A CN 202010838819 A CN202010838819 A CN 202010838819A CN 112035571 A CN112035571 A CN 112035571A
- Authority
- CN
- China
- Prior art keywords
- hash value
- target hash
- data operation
- data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000006870 function Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data synchronization method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a plurality of data operation records in a source database; determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in the table; and distributing the data operation records with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the data operation records so as to enable the threads to execute corresponding data synchronization tasks in parallel. By the technical scheme of the embodiment of the invention, the problem that the concurrent processing of the data operation records in the same table cannot be carried out in the prior art can be solved, and the data synchronization efficiency is improved.
Description
Technical Field
Embodiments of the present invention relate to computer technologies, and in particular, to a data synchronization method, apparatus, device, and storage medium.
Background
With the rapid development of computer technology, data operations, such as write operations and update operations, are usually performed on data, and corresponding operations are performed on a target database based on data operation records in a source database to implement data synchronization.
In the prior art, data operation records in a source database are generally divided based on table dimensions, and the data operation records with the same table name are sent to the same thread for processing according to a data operation sequence in the source database, so as to ensure the accuracy of data synchronization.
However, in the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art:
the existing data synchronization mode is that all data operation records in the same table are processed and executed by the same thread sequence, so that the data operation records in the same table cannot be processed concurrently, and the data synchronization efficiency is greatly reduced.
Disclosure of Invention
Embodiments of the present invention provide a data synchronization method, apparatus, device, and storage medium, so as to solve a problem that concurrent processing cannot be performed on data operation records in the same table in the prior art, and improve data synchronization efficiency.
In a first aspect, an embodiment of the present invention provides a data synchronization method, including:
acquiring a plurality of data operation records in a source database;
determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in a table;
and distributing each data operation record with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the plurality of data operation records, so that each thread executes corresponding data synchronization tasks in parallel.
In a second aspect, an embodiment of the present invention further provides a data synchronization apparatus, including:
the data operation record acquisition module is used for acquiring a plurality of data operation records in the source database;
the target hash value determining module is used for determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in a table;
and the data processing module is used for distributing each data operation record with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the plurality of data operation records so as to enable each thread to execute corresponding data synchronization tasks in parallel.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data synchronization method as provided by any of the embodiments of the invention.
In a fourth aspect, the embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data synchronization method provided in any embodiment of the present invention.
The embodiment of the invention has the following advantages or beneficial effects:
the target hash value corresponding to each data operation record is determined according to the table name information and the identification field information in each data operation record in the source database, so that the data operation records with different identification field information in the same table are distinguished by the target hash value. Based on the target hash value and the recording sequence corresponding to the multiple data operation records, distributing the data operation records with the same target hash value to the same thread according to the recording sequence, so that the threads execute corresponding data synchronization tasks in parallel, and distributing the data operation records with different identification field information in the same table to different threads while ensuring the accuracy of data synchronization, so that the multiple threads can be used for performing concurrent processing on the data operation records in the same table, and the data synchronization efficiency is greatly improved.
Drawings
Fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is an exemplary distribution of a record of data operations according to an embodiment of the present invention;
fig. 3 is a flowchart of a data synchronization method according to a second embodiment of the present invention;
fig. 4 is a distribution example of a data operation record according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of a data synchronization method according to an embodiment of the present invention, which is applicable to a case where data operation records in a source database are synchronized to a target database. The method may be performed by a data synchronization apparatus, which may be implemented by software and/or hardware, integrated in a device having a data processing function. As shown in fig. 1, the method specifically includes the following steps:
and S110, acquiring a plurality of data operation records in the source database.
Wherein the source database may be a database for storing the generated raw data. Each data operation record may refer to data information for each data operation, which may correspond to a complete row of information in the table. For example, when a row of information in the table is updated, the updated row of information may be recorded as a data operation. Alternatively, when a new line of information is written into the table, the written line of information may be recorded as a data operation. It should be noted that, multiple data operations may be performed on the same row of data in the table, so that the row may correspond to multiple data operation records. For example, a certain row in the table is subjected to multiple data updating operations, so that multiple data operation records corresponding to the row exist in the source database.
Specifically, the binary log may be pulled from the source database, and the pulled binary log may be analyzed to obtain each analyzed data operation record in sequential arrangement, where data of each table in the target database are mixed together at this time. It should be noted that the data operation records are sequentially generated according to the data operation sequence, so that data synchronization can be performed based on the sequence of the plurality of data operation records in the source database, and the ordering of the data operation and the accuracy of the data synchronization are ensured.
S120, determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in the table.
The table name information may refer to a field value of a table name in a target database to which the data operation record is directed. The same table can correspond to a plurality of data operation records. The identification field information may be a field value for distinguishing different data operation records corresponding to the same table. For example, the identification field information may include: primary key information or unique key information in the table. The primary key information may be used to uniquely identify each piece of data, i.e., each row of data, in the table. The primary key information may include one or more fields. For example, the primary key information may refer to a self-growth field value. The unique key information may refer to field values in the table that are never repeated to avoid adding duplicate data. A table has only one primary key, but may have one or more unique keys.
Illustratively, the table structure of a database table may be as follows:
in the table structure, the Fid in the t _ example _ info table is an autonomous key, and the Fname is a unique key, so that the Fid or the Fname can be set as an identification field.
Specifically, in this embodiment, the primary key information or a unique key information in the table may be used as the identification field information to distinguish the data operation records. The same identification field can be set for different tables in the embodiment to simplify the data synchronization operation; different identification fields may also be provided to improve flexibility in data synchronization. For example, a corresponding identification field may be configured for each table in the target database, the table name may be used as Key information Key, the configured corresponding identification field may be used as Value information Value, and Key Value pair information is created, so that, when data is synchronized, table name information may be obtained from a data operation record for each data operation record, and the table name information may be queried in the Key Value pair information as Key to obtain a corresponding identification field, so that identification field information corresponding to the identification field may be obtained from the data operation record, thereby increasing configuration flexibility and diversity of data synchronization.
When the table name information and the identification field information in each data operation record are obtained, hash operation can be performed on the table name information and the identification field information to determine a target hash value corresponding to each data operation record. The table name information and the identification field information of the data operation records corresponding to the same row of data in the same table in the target database are the same, so that the corresponding target hash values are also the same, and therefore, when the target hash values are the same, it can be obtained that the data operation records are processed aiming at the same row of data (namely, the same identification field information) in the same table.
Exemplarily, S120 may include: determining a first hash value corresponding to the table name information and a second hash value corresponding to the identification field information in each data operation record; and determining a target hash value corresponding to the data operation record based on the first hash value and the second hash value.
Specifically, a first hash value corresponding to the table name information and a second hash value corresponding to the identification field information in each data operation record may be determined based on an existing hash algorithm. For each data operation record, the product of the first hash value and the second hash value can be used as a target hash value corresponding to the data operation record, so that whether the table name information and the identification field information in the data operation record are the same or not can be judged more conveniently by comparing whether the target hash values are the same or not, and the data synchronization efficiency is further improved.
And S130, distributing the data operation records with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the data operation records, so that the threads execute corresponding data synchronization tasks in parallel.
The record sequence corresponding to the plurality of data operation records may refer to a sequence of the plurality of data operation records stored in the source database, and may be used to represent the data operation sequence. The threads may be pre-configured to process the distributed data operation records. The data synchronization task may be a task in which the thread sequentially writes the distributed data operation records into corresponding tables in the target database, so that the data can be synchronized from the source database to the target database. In this embodiment, a certain number of threads may be set in advance based on device resources and service requirements, so that multiple threads may execute corresponding data synchronization tasks in parallel.
Specifically, the target hash values corresponding to each data operation record may be compared, and all data operation records having the same target hash value are distributed to the same thread according to the original record sequence, so that the data operation records having the same table name information and the same identification field information are processed by the same thread, that is, a row of data in one table in the target database is processed by the same thread. The data operation records with the same table name information but different identification field information correspond to different target hash values, so that the data operation records corresponding to other rows of data in the same table can be distributed to another thread for processing, and the data operation records with different identification field information in the same table can be distributed to different threads. Through the concurrent execution of a plurality of threads, the data operation records in the same table can be concurrently processed, the data synchronization efficiency is greatly improved, the condition of data synchronization delay caused by the fact that the concurrent processing cannot be carried out in the prior art can be avoided, and the accuracy of data synchronization is ensured.
According to the technical scheme of the embodiment, the target hash value corresponding to each data operation record is determined according to the table name information and the identification field information in each data operation record in the source database, so that the data operation records with different identification field information in the same table are distinguished by using the target hash value. Based on the target hash value and the recording sequence corresponding to the multiple data operation records, distributing the data operation records with the same target hash value to the same thread according to the recording sequence, so that the threads execute corresponding data synchronization tasks in parallel, and distributing the data operation records with different identification field information in the same table to different threads while ensuring the accuracy of data synchronization, so that the data operation records in the same table can be processed concurrently by utilizing multiple threads, and the data synchronization efficiency is greatly improved.
On the basis of the foregoing technical solution, determining a hash value corresponding to the data operation record based on the first hash value and the second hash value may include: and adding the first hash value and the second hash value, and determining the obtained addition result as a target hash value corresponding to the data operation record.
Specifically, for each data operation record, the first hash value and the second hash value may be added, and the addition result is used as a target hash value corresponding to the data operation record, so as to avoid the target hash value being too large, so that the target hash value is more convenient to compare, and further improve the data synchronization efficiency.
On the basis of the above technical solution, S130 may include: determining the corresponding relation between each target hash value and each thread; distributing the data operation records to corresponding threads one by one according to the recording sequence corresponding to the data operation records and on the basis of the target hash value corresponding to each data operation record and the corresponding relation; and writing the data operation records in each thread into corresponding tables in the target database one by concurrently executing the threads.
Specifically, the corresponding relationship between the target hash value and the thread may be determined based on the obtained target hash values and the preset thread number, and the thread corresponding to each target hash value is obtained, so that the data operation records may be distributed to the thread corresponding to the target hash value one by one according to the recording sequence corresponding to the plurality of data operation records, and the data operation records in each thread are also sorted according to the original recording sequence, so that each thread may execute each data operation record in sequence according to the recording sequence, thereby ensuring the orderliness and consistency of data operation, and further ensuring the accuracy of data synchronization. After all the data operation records are distributed to the corresponding threads, the threads are executed concurrently, so that each thread sequentially executes the received data operation records, namely the data operation records are sequentially written into the corresponding tables in the target database, and corresponding data synchronization tasks are completed. For example, each thread may sequentially store the received data operation records into the same queue, so that each data operation record is sequentially written into a corresponding table in the target database based on the queue, thereby ensuring the ordering of the data operation. Because the data operation records of different identification field information in the same table can be distributed to different threads, the data operation records in the same table can be processed concurrently by using a plurality of threads, the data synchronization efficiency is improved, and the data operation records corresponding to the same identification field information in the same table are executed by the same thread according to the original data operation sequence, so that the accuracy of data synchronization is also ensured.
For example, determining the correspondence between each target hash value and each thread may include: if the number of the target hash values is less than or equal to the number of the preset threads, selecting threads with the same number as the target hash values from all the threads, and determining the corresponding relation between each target hash value and each selected thread; and if the number of the target hash values is larger than the preset thread number, dividing each target hash value to obtain a target hash value set of the preset thread number, and determining the corresponding relation between each target hash value and each thread based on each target hash value set.
Specifically, the number of different obtained target hash values may be counted, the number of the target hash values is compared with the number of preset threads, and if the number of the target hash values is less than or equal to the number of the preset threads, it is indicated that the data operation record corresponding to each target hash value may be processed by a single thread, that is, the target hash values corresponding to the data operation records processed by each thread are the same, at this time, threads having the same number as the number of the target hash values may be randomly selected from the thread pool, and a corresponding relationship between each target hash value and each thread may be randomly established, so as to ensure that the target hash values and the threads correspond one to one. By distributing based on the one-to-one correspondence relationship, each thread can be only used for processing the data operation records corresponding to the same row of the same table, so that the data synchronization efficiency can be further improved, and the condition of data synchronization delay is avoided. If the number of the target hash values is larger than the number of the preset threads, it indicates that the data operation records with different target hash values cannot be processed by using the threads, at this time, all the target hash values can be divided to obtain the target hash value sets with the same number as the number of the preset threads, and a one-to-one correspondence relationship between the target hash value sets and the threads can be randomly established, so that a many-to-one relationship between each target hash value and the threads is obtained based on each target hash value contained in the target hash value sets, that is, one thread can correspond to one or more target hash values. By distributing based on the many-to-one relationship, all data operation records can be effectively processed by using a limited number of threads, and further, the diversity of the scene applicability is improved.
For example, dividing each target hash value to obtain a set of target hash values for a preset number of threads may include: dividing each target hash value by the number of preset threads to obtain a remainder corresponding to each target hash value; each target hash value having the same remainder is divided into a set of target hash values.
Specifically, remainder taking processing may be performed on each target hash value based on the number of preset threads to obtain a remainder corresponding to each target hash value, and all target hash values having the same remainder are divided into one group to obtain one target hash value set, so that each target hash value may be effectively divided in a manner of remainder taking processing.
For example, fig. 2 shows an example of distribution of a data operation record. As shown in FIG. 2, assume that there are two tables in the target database, Table A and Table B. Tables a and B represent table name information. 00. 02, 08, 01 and 12 each represent identification field information. Fig. 2 shows 10 data operation records, which are arranged in sequence according to the data operation order. There are two table a00 records in fig. 2 that are used to characterize that there are two data operation records for which the table name information and identification field information are the same. By determining the target hash value corresponding to each data operation record, 5 different target hash values can be obtained. If it is assumed that only two threads exist in the thread pool, by dividing the 5 target hash values, the target hash values corresponding to the records in table a00, the target hash values corresponding to the records in table a02, and the target hash values corresponding to the records in table B12 may be divided into one group, that is, the data operation records corresponding to the three target hash values need to be distributed to the same thread, that is, to thread 1. And dividing the target hash values corresponding to the records in table a08 and the target hash values corresponding to the records in table B01 into another group, that is, indicating that the data operation records corresponding to the two target hash values need to be distributed to another thread, that is, to thread 2, so that a larger number of data operation records can be processed by using a limited number of threads, and the diversity of the application of the scene is improved.
Example two
Fig. 3 is a flowchart of a data synchronization method according to a second embodiment of the present invention, where on the basis of the foregoing embodiments, the present embodiment describes in detail the determination of the target hash value set when the number of the target hash values is greater than the number of the preset threads. Wherein explanations of the same or corresponding terms as those of the above embodiments are omitted.
Referring to fig. 3, the data synchronization method provided in this embodiment specifically includes the following steps:
s310, acquiring a plurality of data operation records in the source database.
S320, determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in the table.
S330, if the number of the target hash values is larger than the number of the preset threads, updating each target hash value based on a preset load balancing function with idempotency, and obtaining the updated target hash value.
The preset load balancing function can be a function for balancing load, so that the distribution of the target hash value is more uniform, the quantity difference of the data operation records received by each thread is smaller, and the inclination of output data is avoided. Idempotency may refer to the fact that the arguments obtained after the same arguments (i.e., target hash values) are passed in are consistent and unique, thereby avoiding disrupting the ordering of data operations. For example, the pre-set load balancing function may be, but is not limited to, a Murmur3 non-cryptographic hash function.
Specifically, when the number of the target hash values is greater than the preset number of threads, each thread may process data operation records corresponding to one or more target hash values, so that data operation records in one or some threads are excessive, and data tilt may be caused. For this situation, in this embodiment, each target hash value may be used as a parameter by using a preset load balancing function with idempotency, and the obtained operation result is used as an updated target hash value through the operation of the preset load balancing function, so that the updated target hash value is more uniformly distributed.
S340, dividing each updated target hash value to obtain a target hash value set with a preset thread number.
Specifically, the updated target hash values are distributed more uniformly, so that the difference between the numbers of the target hash values included in the target hash value set obtained after division is smaller.
And S350, determining the corresponding relation between each target hash value and each thread based on each target hash value set.
Specifically, the difference between the numbers of the target hash values included in each target hash value set is small, so that the difference between the numbers of the target hash values corresponding to each thread is small, and the condition of load balancing can be further ensured.
And S360, distributing the data operation records to corresponding threads one by one according to the data operation sequence corresponding to the data operation records and on the basis of the target hash value and the corresponding relation corresponding to each data operation record.
Specifically, based on the corresponding relationship determined in S350, the data operation records received by each thread can be relatively balanced, so that the problem of data skew caused by uneven distribution of the target hash value can be solved, the problem of data synchronization delay caused by data skew is avoided, and the data synchronization efficiency can be further improved.
For example, fig. 4 shows an example of distribution of a data manipulation record. As shown in FIG. 4, assume that there are two tables in the target database, Table A and Table B. Tables a and B represent table name information. 00. 02, 08, 01 and 12 each represent identification field information. Fig. 4 shows 10 data operation records, which are arranged in sequence according to the data operation order. By determining the target hash value corresponding to each data operation record, 5 different target hash values can be obtained. If it is assumed that only two threads exist in the thread pool, that is, the number of the target hash values is greater than the number of the threads, each target hash value can be updated based on a preset load balancing function with idempotency, to obtain 5 target hash values which are more uniformly distributed after updating, and by dividing the 5 updated target hash values, the target hash values corresponding to the records in table a00 and the target hash values corresponding to the records in table a02 can be divided into one group, that is, it is indicated that the data operation records corresponding to the two updated target hash values need to be distributed to the same thread, that is, to thread 1. And dividing the target hash values corresponding to the records in table B12, table a08, and table B01 into another group, that is, indicating that the data operation records corresponding to the three updated target hash values need to be distributed to another thread, that is, to thread 2, so that, compared with the distribution result in fig. 2, the target hash values are more uniformly distributed by using the preset load balancing function in fig. 4, thereby ensuring load balancing and avoiding data skew.
And S370, writing the data operation records in each thread into corresponding tables in the target database one by concurrently executing the threads.
The technical scheme of this embodiment, when the quantity through at the target hash value is greater than and predetermines the thread quantity, based on the predetermined load balancing function that has idempotent nature, update every target hash value, thereby can make the distribution of the target hash value after the update more even, and divide based on the target hash value after the update, make the difference of the target hash value quantity that the target hash value set that obtains after dividing includes less, and then make the data operation record that every thread received comparatively balanced, thereby can solve the condition that target hash value distributes inhomogeneously and leads to the data skew, and then avoid leading to the condition that data synchronization delays because of the data skew, thereby can further improve data synchronization efficiency.
The following is an embodiment of a data synchronization apparatus provided in an embodiment of the present invention, and the apparatus and the data synchronization method of the foregoing embodiments belong to the same inventive concept, and details that are not described in detail in the embodiment of the data synchronization apparatus may refer to the embodiment of the data synchronization method.
EXAMPLE III
Fig. 5 is a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention, which is applicable to a case where data operation records in a source database are synchronized to a target database, and the apparatus specifically includes: a data operation record obtaining module 510, a target hash value determining module 520, and a data processing module 530.
The data operation record obtaining module 510 is configured to obtain multiple data operation records in a source database; a target hash value determining module 520, configured to determine a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, where the identification field information is used to uniquely identify one data operation record in the table; and the data processing module 530 is configured to distribute, based on the target hash value and the recording order corresponding to the multiple data operation records, the multiple data operation records having the same target hash value to the same thread according to the recording order, so that the threads execute corresponding data synchronization tasks in parallel.
Optionally, the target hash value determining module 520 includes:
the hash value determining unit is used for determining a first hash value corresponding to the table name information in each data operation record and a second hash value corresponding to the identification field information;
and the target hash value determining unit is used for determining a target hash value corresponding to the data operation record based on the first hash value and the second hash value.
Optionally, the target hash value determining unit is specifically configured to: and adding the first hash value and the second hash value, and determining the obtained addition result as a target hash value corresponding to the data operation record.
Optionally, the data processing module 530 includes:
a correspondence determining unit configured to determine a correspondence between each target hash value and each thread;
the data operation record distribution unit is used for distributing the data operation records to corresponding threads one by one according to the recording sequence corresponding to the data operation records and on the basis of the target hash value and the corresponding relation corresponding to each data operation record;
and the data operation record writing unit is used for writing the data operation records in each thread into corresponding tables in the target database one by executing the threads concurrently.
Optionally, the correspondence determining unit includes:
the first determining subunit is used for selecting threads with the same number as the target hash values from the threads and determining the corresponding relation between each target hash value and each selected thread if the number of the target hash values is less than or equal to the number of the preset threads;
and the second determining subunit is used for dividing each target hash value to obtain a target hash value set of the preset thread number if the number of the target hash values is greater than the preset thread number, and determining the corresponding relation between each target hash value and each thread based on each target hash value set.
Optionally, the second determining subunit is specifically configured to: dividing each target hash value by the number of preset threads to obtain a remainder corresponding to each target hash value; each target hash value having the same remainder is divided into a set of target hash values.
Optionally, the second determining subunit is further specifically configured to: updating each target hash value based on a preset load balancing function with idempotency to obtain an updated target hash value; and dividing each updated target hash value to obtain a target hash value set with a preset thread number.
The data synchronization device provided by the embodiment of the invention can execute the data synchronization method provided by any embodiment of the invention, and has the corresponding functional module and beneficial effect of executing the data synchronization method.
It should be noted that, in the embodiment of the data synchronization apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
Example four
Fig. 6 is a schematic structural diagram of an apparatus according to a fourth embodiment of the present invention. Fig. 6 illustrates a block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention. The device 12 shown in fig. 6 is only an example and should not bring any limitations to the functionality and scope of use of the embodiments of the present invention.
As shown in FIG. 6, device 12 is in the form of a general purpose computing device. The components of device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. Device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 6, and commonly referred to as a "hard drive"). Although not shown in FIG. 6, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, to implement a data synchronization method provided by the embodiment of the present invention, the method includes:
acquiring a plurality of data operation records in a source database;
determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in the table;
and distributing the data operation records with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the data operation records so as to enable the threads to execute corresponding data synchronization tasks in parallel.
Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data synchronization method provided by any embodiment of the present invention.
EXAMPLE five
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a data synchronization method as provided in any of the embodiments of the present invention, the method comprising:
acquiring a plurality of data operation records in a source database;
determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in the table;
and distributing the data operation records with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the data operation records so as to enable the threads to execute corresponding data synchronization tasks in parallel.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer-readable storage medium may be, for example but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It will be understood by those skilled in the art that the modules or steps of the invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and optionally they may be implemented by program code executable by a computing device, such that it may be stored in a memory device and executed by a computing device, or it may be separately fabricated into various integrated circuit modules, or it may be fabricated by fabricating a plurality of modules or steps thereof into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of data synchronization, comprising:
acquiring a plurality of data operation records in a source database;
determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in a table;
and distributing each data operation record with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the plurality of data operation records, so that each thread executes corresponding data synchronization tasks in parallel.
2. The method of claim 1, wherein determining the target hash value corresponding to each of the data operation records according to table name information and identification field information in each of the data operation records comprises:
determining a first hash value corresponding to table name information in each data operation record and a second hash value corresponding to identification field information;
and determining a target hash value corresponding to the data operation record based on the first hash value and the second hash value.
3. The method of claim 2, wherein determining the target hash value corresponding to the data operation record based on the first hash value and the second hash value comprises:
and adding the first hash value and the second hash value, and determining an obtained addition result as a target hash value corresponding to the data operation record.
4. The method of claim 1, wherein distributing the data operation records with the same target hash value to the same thread according to the recording order based on the target hash value and the recording order corresponding to the data operation records, so that the threads execute corresponding data synchronization tasks in parallel, comprises:
determining a corresponding relation between each target hash value and each thread;
distributing the data operation records to corresponding threads one by one according to the recording sequence corresponding to the data operation records and based on the target hash value corresponding to each data operation record and the corresponding relation;
and writing the data operation records in each thread into corresponding tables in the target database one by concurrently executing the threads.
5. The method of claim 4, wherein determining a correspondence between each of the target hash values and each of the threads comprises:
if the number of the target hash values is less than or equal to the number of preset threads, selecting threads with the same number as the target hash values from all the threads, and determining the corresponding relation between each target hash value and each selected thread;
if the number of the target hash values is larger than the number of preset threads, dividing each target hash value to obtain a target hash value set of the number of the preset threads, and determining the corresponding relation between each target hash value and each thread based on each target hash value set.
6. The method according to claim 5, wherein dividing each target hash value to obtain a set of target hash values for a predetermined number of threads comprises:
dividing each target hash value by the number of the preset threads to obtain a remainder corresponding to each target hash value;
each target hash value having the same remainder is divided into a set of target hash values.
7. The method according to claim 5, wherein dividing each target hash value to obtain the set of target hash values for the preset number of threads comprises:
updating each target hash value based on a preset load balancing function with idempotency to obtain an updated target hash value;
and dividing each updated target hash value to obtain a target hash value set of the preset thread number.
8. A data synchronization apparatus, comprising:
the data operation record acquisition module is used for acquiring a plurality of data operation records in the source database;
the target hash value determining module is used for determining a target hash value corresponding to each data operation record according to the table name information and the identification field information in each data operation record, wherein the identification field information is used for uniquely identifying one data operation record in a table;
and the data processing module is used for distributing each data operation record with the same target hash value to the same thread according to the recording sequence based on the target hash value and the recording sequence corresponding to the plurality of data operation records so as to enable each thread to execute corresponding data synchronization tasks in parallel.
9. An apparatus, characterized in that the apparatus comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data synchronization method as recited in any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data synchronization method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010838819.0A CN112035571A (en) | 2020-08-19 | 2020-08-19 | Data synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010838819.0A CN112035571A (en) | 2020-08-19 | 2020-08-19 | Data synchronization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112035571A true CN112035571A (en) | 2020-12-04 |
Family
ID=73577685
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010838819.0A Withdrawn CN112035571A (en) | 2020-08-19 | 2020-08-19 | Data synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035571A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685182A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113204602A (en) * | 2021-05-07 | 2021-08-03 | 星环信息科技(上海)股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113672624A (en) * | 2021-08-25 | 2021-11-19 | 中汽创智科技有限公司 | Data processing method, system and storage medium |
| CN113760907A (en) * | 2021-08-02 | 2021-12-07 | 紫金诚征信有限公司 | Data uniqueness identification method in database |
| CN113901141A (en) * | 2021-10-11 | 2022-01-07 | 京信数据科技有限公司 | Distributed data synchronization method and system |
| CN114679503A (en) * | 2022-03-24 | 2022-06-28 | 长桥有限公司 | Market data processing method, system and equipment |
| CN115858309A (en) * | 2022-12-30 | 2023-03-28 | 北京百度网讯科技有限公司 | Data monitoring method and device for distributed system and electronic equipment |
| CN116577568A (en) * | 2023-07-13 | 2023-08-11 | 中国气象科学研究院 | Lightning positioning method and device based on hash value characteristics |
-
2020
- 2020-08-19 CN CN202010838819.0A patent/CN112035571A/en not_active Withdrawn
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685182A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113204602A (en) * | 2021-05-07 | 2021-08-03 | 星环信息科技(上海)股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113204602B (en) * | 2021-05-07 | 2023-08-01 | 星环信息科技(上海)股份有限公司 | Data processing method, device, equipment and storage medium |
| CN113760907A (en) * | 2021-08-02 | 2021-12-07 | 紫金诚征信有限公司 | Data uniqueness identification method in database |
| CN113672624A (en) * | 2021-08-25 | 2021-11-19 | 中汽创智科技有限公司 | Data processing method, system and storage medium |
| CN113901141B (en) * | 2021-10-11 | 2022-08-05 | 京信数据科技有限公司 | Distributed data synchronization method and system |
| CN113901141A (en) * | 2021-10-11 | 2022-01-07 | 京信数据科技有限公司 | Distributed data synchronization method and system |
| CN114679503A (en) * | 2022-03-24 | 2022-06-28 | 长桥有限公司 | Market data processing method, system and equipment |
| CN114679503B (en) * | 2022-03-24 | 2024-08-06 | 长桥有限公司 | Quotation data processing method, system and equipment |
| CN115858309A (en) * | 2022-12-30 | 2023-03-28 | 北京百度网讯科技有限公司 | Data monitoring method and device for distributed system and electronic equipment |
| CN115858309B (en) * | 2022-12-30 | 2023-12-22 | 北京百度网讯科技有限公司 | Data monitoring method and device for distributed system and electronic equipment |
| CN116577568A (en) * | 2023-07-13 | 2023-08-11 | 中国气象科学研究院 | Lightning positioning method and device based on hash value characteristics |
| CN116577568B (en) * | 2023-07-13 | 2023-10-27 | 中国气象科学研究院 | Lightning positioning method and device based on hash value characteristics |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112035571A (en) | Data synchronization method, device, equipment and storage medium | |
| CN108537543B (en) | Parallel processing method, device, equipment and storage medium for blockchain data | |
| US10545762B2 (en) | Independent mapping of threads | |
| US9465840B2 (en) | Dynamically indentifying and preventing skewed partitions in a shared-nothing database | |
| US11636107B2 (en) | Database management system, computer, and database management method | |
| US10496659B2 (en) | Database grouping set query | |
| US20090248596A1 (en) | Configuration information management apparatus, configuration information management program, and configuration information management method | |
| US9684689B2 (en) | Distributed parallel processing system having jobs processed by nodes based on authentication using unique identification of data | |
| US9665532B2 (en) | Performing synchronized collective operations over multiple process groups | |
| CN110673959A (en) | System, method and apparatus for processing tasks | |
| CN111611249A (en) | Data management method, device, equipment and storage medium | |
| US11048557B2 (en) | Methods and modules relating to allocation of host machines | |
| US10048991B2 (en) | System and method for parallel processing data blocks containing sequential label ranges of series data | |
| CN109165723B (en) | Method and apparatus for processing data | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| US20140236977A1 (en) | Mapping epigenetic surprisal data througth hadoop type distributed file systems | |
| CN111444148A (en) | Data transmission method and device based on MapReduce | |
| WO2023071566A1 (en) | Data processing method and apparatus, computer device, computer-readable storage medium, and computer program product | |
| US10579428B2 (en) | Data token management in distributed arbitration systems | |
| CN112148705A (en) | Data migration method and device | |
| US8775776B2 (en) | Hash table using hash table banks | |
| US9137276B2 (en) | Managing message distribution in a networked environment | |
| CN109947613B (en) | File reading test method and device | |
| US20140095531A1 (en) | Dynamic output selection using highly optimized data structures | |
| US11436261B2 (en) | Shuffle-less reclustering of clustered tables |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201204 |