CN109739936A - Method of data synchronization, system, server and computer readable storage medium - Google Patents
Method of data synchronization, system, server and computer readable storage medium Download PDFInfo
- Publication number
- CN109739936A CN109739936A CN201910063590.5A CN201910063590A CN109739936A CN 109739936 A CN109739936 A CN 109739936A CN 201910063590 A CN201910063590 A CN 201910063590A CN 109739936 A CN109739936 A CN 109739936A
- Authority
- CN
- China
- Prior art keywords
- data
- tables
- interim table
- variation
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The application provides a kind of method of data synchronization, comprising: when detecting the first tables of data, there are delta datas, and when needing delta data being synchronized to the second tables of data, judge whether data variation type is that data update or data are deleted；If so, interim table is written in delta data；Interim table is merged with the second tables of data, it is synchronous to complete data.When being updated there are data or data are deleted, variation record is batch insert action when interim table is written from the first tables of data, without making update or delete processing to actual data content according to variation record, different from variation directly to be recorded to the update one by one or deletion movement of the second tables of data of write-in, improve the speed that data update or data are deleted, the storage performance of database is improved, solves the problems, such as that more new data when data are synchronous, deletion data efficiency are low.The application also provides data synchronous system, computer readable storage medium and the server between database, has identical beneficial effect.
Description
Technical field
This application involves database field, in particular to a kind of method of data synchronization, data synchronous system, a kind of computer
Readable storage medium storing program for executing and a kind of server.
Background technique
Existing incremental data is synchronous, and there are many solutions, and there are commonly timestamp mode, trigger mode, log solutions
Analysis mode and Oracle delta data capture (changed data capture, CDC) mode.It is in a manner of Oracle CDC
Example, can capture changed data in a period of time, and data insertion, the record for updating, deleting can be saved.Pass through parsing
These records, realize the synchronization of incremental data.
However, when source library incremental data is larger, for the not high database of some write performances, such as GBase,
The analytical database such as GreenPlum does not support batch updating, batch to delete, can only update one by one, delete data, directly
The synchronous efficiency of data is influenced, and then reduces the storage performance of database.
In addition, the database of batch processing is supported for MySQL, Oracle etc., the main side for using precompile SQL statement
Formula is realized batch updating, is deleted.But when carrying out data update, deleting, need to inquire the data with existing in database, if frequency
It is updated, deletes or when purpose table data volume is larger, the efficiency of increment synchronization also be will be greatly reduced numerously.
Summary of the invention
The purpose of the application is to provide a kind of method of data synchronization, data synchronous system, a kind of computer-readable storage medium
Matter and a kind of server, the low efficiency for solving the problems, such as more new data when data are synchronous between existing database, deleting data.
In order to solve the above technical problems, the application provides a kind of method of data synchronization, specific technical solution is as follows:
When detecting the first tables of data, there are delta datas, and need the delta data being synchronized to the second tables of data
When, judge whether data variation type is that data update and/or data are deleted；
If so, interim table is written in the variation record of the delta data；Wherein, the interim table and second number
It is identical according to table table structure；
The interim table is merged with second tables of data, in second tables of data according to the variation recording synchronism
After the delta data, it is synchronous to complete data.
Wherein, the interim table includes the first interim table and the second interim table；
When the data variation type is that data update, data are updated into corresponding first variation record write-in described the
One interim table；
When the data variation type is that data are deleted, data are deleted into corresponding second variation record write-in described the
Two interim tables；
When the data variation type is that data update and data are deleted, data are updated into corresponding first variation and are recorded
The described first interim table is written, data is deleted into corresponding second variation record, the second interim table is written.
It is wherein, described that data are updated into the corresponding first variation record write-in first interim table, comprising:
Data are updated into corresponding first variation record batch and are inserted into the described first interim table.
It is wherein, described that data are deleted into the corresponding second variation record write-in second interim table, comprising:
Data are updated into corresponding second variation record batch and are inserted into the described second interim table.
Wherein, before interim table being written in the variation record of the delta data, further includes:
The interim table is generated, and is associated with the interim table and second tables of data.
Wherein, the interim table is merged with second tables of data, completion data, which synchronize, includes:
The delta data batch is incorporated to described second using Sql according to the variation record in the interim table
It is synchronous to complete data for tables of data.
Wherein, the interim table is merged with second tables of data, after completion data are synchronous further include:
Generate the synchronous success message of data.
The application also provides the data synchronous system between a kind of database, comprising:
Judgment module, for that ought detect the first tables of data, there are delta datas, and need the delta data is synchronous
When to the second tables of data, judge whether data variation type is that data update or data are deleted；
Writing module, for interim table to be written in the variation record of the delta data；Wherein, the interim table with it is described
Second tables of data table structure is identical；
It is synchronous to complete data for merging the interim table with second tables of data for synchronization module.
Wherein, the interim table includes the first interim table and the second interim table；
When the data variation type is that data update, the write module, which is used to updating data into corresponding first, to be become
Change record and the first interim table is written；
When the data variation type is that data are deleted, the write module, which is used to deleting data into corresponding second, to be become
Change record and the second interim table is written；
When the data variation type updates for data and data are deleted, the write module is used for data update pair
The first interim table is written in the first variation record answered, and data are deleted corresponding second variation record write-in described second and are faced
When table.
Wherein, the write module is inserted into described the particularly for data are updated corresponding first variation record batch
The module of one interim table.
Wherein, the write module is inserted into described the particularly for data are updated corresponding second variation record batch
The module of two interim tables.
Wherein, the data synchronous system further include:
Temp table generation module for generating the interim table, and is associated with the interim table and second tables of data.
Wherein, the synchronization module utilizes Sql by institute particularly for according to the variation record in the interim table
It states delta data batch to be incorporated to second tables of data, completes the synchronous module of data.
Wherein, the data synchronous system further include:
Message generating module, for generating the synchronous success message of data.
The application also provides a kind of computer readable storage medium, is stored thereon with computer program, the computer journey
The step of method of data synchronization as described above is realized when sequence is executed by processor.
The application also provides a kind of server, including memory and processor, has computer program in the memory,
The step of processor realizes method of data synchronization as described above when calling the computer program in the memory.
A kind of method of data synchronization provided herein, comprising: when detecting the first tables of data, there are delta datas, and
When needing the delta data being synchronized to the second tables of data, judge whether data variation type is that data update or data are deleted
It removes；If so, interim table is written in the variation record of the delta data；The interim table and second tables of data are closed
And it is synchronous to complete data.
Since the efficiency far of data insertion is higher than data update and deletes, particularly with the analytic types database such as GBase
Do not support batch updating, delete data, the application there are data update or data delete when, first by data record write-in with
The interim table that second tables of data can directly merge is inserted into dynamic in batches when interim table is written from the first tables of data in variation record
Work, without making update or delete processing to actual data content according to variation record, being different from directly will variation record write-in
The update one by one or deletion of second tables of data act, and improve the speed that data update or data are deleted, improve database
Storage performance, solve the problems, such as inefficiency in normal data synchronization.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of application for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow chart of method of data synchronization provided by the embodiment of the present application；
Data synchronous system block diagram of the Fig. 2 between a kind of database provided by the embodiment of the present application；
Fig. 3 is a kind of structural schematic diagram of server provided by the embodiment of the present application；
Fig. 4 is the structural schematic diagram of another kind server provided by the embodiment of the present application.
Specific embodiment
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall in the protection scope of this application.
Referring to FIG. 1, Fig. 1 is a kind of flow chart of method of data synchronization provided by the embodiment of the present application, the data are same
One step process includes:
S101: when detecting the first tables of data, there are delta datas, and need for the delta data to be synchronized to the second number
When according to table, judge whether data variation type is that data update or data are deleted；If so, into S102；
This step is intended to detect whether that there are delta datas, and judge the delta data change type whether be data more
New or data are deleted.It is readily apparent that the object of detection is database, more specifically, detection is number in database
According to table.Database synchronization between database requires the data between each database identical, it is therefore necessary to each tables of data into
Row detection, to determine whether there is delta data.
Here the first tables of data and the second tables of data usually belongs to two databases, i.e., and in first database
One tables of data is there are delta data, and first database needs for delta data to be synchronized to the second database at this time, and the second data
Table is then tables of data corresponding with the first tables of data in the second database.And it is understood that the first tables of data can be
The tables of data of data variation arbitrarily occurs in first database.Certainly, actual data are synchronous if it exists needs, and can also apply
Data in same database between different data table are synchronous, herein for the specific of the first tables of data and the second tables of data
System is not construed as limiting.
In general, data variation is divided into data insertion, data update and data are deleted, and this step needs to judge the first data
Whether the data variation of table is that data update and/or data are deleted.It should be noted that when data are inserted into, it can directly in batches
To the second tables of data, i.e., database supports batch to be inserted into, however does not support to carry out data update or data in bulk for write-in
It deletes.Here database generally refers to analytical database, such as GBase, GreenPlum etc., corresponding, the first number
It can be GBase tables of data, GreenPlum tables of data etc. according to table and the second tables of data.It is, of course, also possible to be other tables of data,
A different citing limits herein.
If data variation is that data update and/or data are deleted, enter S102.If not being that data update or data are deleted
It removes, then usually data are inserted into, then the batch data of insertion directly can be synchronized to the second tables of data at this time.
S102: interim table is written into the variation record of the delta data；
Since when data variation is data update perhaps data deletion, database does not support batch updating or batch to delete
It removes.Therefore interim table is written in the variation record of delta data by the embodiment of the present application.It should be noted that interim table and the second number
It is identical according to table table structure.
It is easily understood that there are steps before this step for this step default:
The interim table is generated, and is associated with the interim table and second tables of data.
That is, needing to generate interim table in advance before this step.It should be noted that the table structure of interim table and the
Two tables of data are identical, and the two needs to be associated.The Table Properties type of interim table is not construed as limiting herein, it is preferred that this faces
When table can for hide table.Such as second tables of data be GBase tables of data when, then interim table be the interim table of GBase.Table structure phase
It is both the basis that interim table merges in subsequent step S103 with the second tables of data.It is understood that being deposited in the second database
In several tables of data, due to the second database and the first database relationship that be in data synchronous, the second tables of data is the second number
According to tables of data corresponding with the first tables of data in library, then there is also not corresponding with the first tables of data naturally in the second database
Third tables of data, in order to avoid data synchronization process gets muddled, it is therefore desirable to which interim table is associated with the second tables of data.It is right
In analytic type database, in the related technology, the batch that the second tables of data cannot achieve data is deleted and batch updating, whenever reception
It needs to execute delete operation after to deletion record perhaps more new record immediately or updates operation, and in data deletion or data
Next record can be executed after update.In addition it is also necessary to explanation, interim table is not necessarily a table, can
With the tables of data set being made of multiple tables of data.When interim table includes multiple tables of data, it is necessary to by each tables of data and the
Two tables of data are associated, guarantee that mistake does not occur for subsequent merging process.
And this step needs that interim table is written delta data into.Although being pointed out that interim table and the second tables of data
Structure is identical, but it is batch insert action that practice change, which records when interim table is written from the first tables of data, directly will be different from
The update one by one or deletion movement of variation record the second tables of data of write-in.Because being actually to record variation when interim table is written
" addition " to interim table, therefore be actually written the process of interim table and data be inserted into it is identical when the second tables of data, can be with
For the insertion of batch.Either data update or data are deleted, this step is by more new record accordingly or to delete note
Record write-in actually is equivalent to increase the writing process of record when " data increase " to interim table, therefore this step is actually big
The process by more new record or deletion record write-in is accelerated greatly.
Specifically, this step can be subdivided into the execution of two steps, the first step first records according to the variation of delta data and generates phase
The sql instruction answered, second step recycle sql instruction that interim table is written in variation record.Here sql refers to data
Interim table is written for that will change record in library instruction.
It should be pointed out that the command content of sql instruction here is made accordingly generally according to different types of delta data
Variation.For example, for data update, so it is easy to understand that at least need to include more new content and update in sql instruction
Location information；Process is deleted for data, at least needs to include the location information that will delete data in sql instruction.It utilizes
Variation record is written to interim table by sql instruction, and actually variation record batch is inserted into interim table, is supported using database
Batch increases this feature, and variation record is first inserted into interim table in bulk.
S103: the interim table is merged with second tables of data, and it is synchronous to complete data.
This step is intended to merge interim table with the second tables of data, and it is synchronous to complete data.Above it is stated that, interim table and
Two tables of data are identical table structure.It can directly merge at this time, the second tables of data is according to variation record to certainly after merging
Body carries out corresponding data update or data are deleted, complete it is synchronous with the data of the first tables of data, realization first database and
Data between second database are synchronous.
S102 it is stated that, either variation record is specially more new record or deletion record, and interim table is being written
When be analogous to data insertion form realize batch processing.Therefore the process interim table merged with the second tables of data
Actually the second tables of data reads variation record, and then is recorded according to variation and realize that data update or data are deleted, and completes number
According to synchronization.
Specifically, this step can usually carry out in memory, can be recorded according to the variation in the interim table
The delta data batch is incorporated to second tables of data using Sql, it is synchronous to complete data.
The embodiment of the present application is synchronized to purpose table it is not necessary that data are directly updated the record deleted with data one by one, but sharp
The batch synchronization to more new record and deletion record is realized with interim table.In fact, deleting process in data update or data
Be utilized database support batch be inserted into this feature of the second tables of data, first will variation record batch write-in to the second data
The identical interim table of table table structure, then realize merging for interim table and the second tables of data.It improves and is inquiring, dividing using database
Performance advantage in analysis movement greatly improves data update, deletion efficiency, improves the effect that data are synchronous between database
Rate.
Based on the above embodiment, as preferred embodiment, S103 can also include: later
Generate the synchronous success message of data.
In the prior art, due to database do not support batch updating or batch delete, one by one will more new record or
After person's deletion record synchronizes, it is required to synchronization or synchronizing for deletion record to this more new record and successfully confirms, with
Determine that this synchronizing process successfully completes.It should be noted that this step is usually processor in interim table and the second tables of data
The data confirmation of synchronization success message generated after merging.It is easily understood that when variation record is from the first tables of data to interim table
Or interim table should generate corresponding failure of data synchronization and believe when merging the abnormal conditions such as mistake with the second tables of data
Breath, for subsequent data second synchronization or database maintenance etc..
And the present embodiment only need interim table and the second tables of data completion synchronize after do one it is subsynchronous successfully confirm, be not necessarily to
Do multiple confirmation.When progress data are synchronous between database, needs, which pass through, largely changes recording synchronism, and the present embodiment
By it is several it is subsynchronous successfully confirmation processes be reduced to only need to carry out one it is subsynchronous successfully confirm, avoid multiple data and synchronize really
Recognize, improves the speed that data update or data are deleted, improve the storage performance of database, solve normal data synchronization
The problem of middle inefficiency.
Based on the above embodiment, as preferred embodiment, interim table may include the first interim table and the second interim table；
When data variation type is that data update, it is interim that data are updated into corresponding first variation record write-in first
Table.Correspondingly, data are updated into corresponding first variation record the first interim table of write-in, it can be with specifically: first more according to data
New corresponding first variation, which records, generates corresponding first Sql instruction, and the first variation is recorded batch further according to the first Sql instruction
It is inserted into the first interim table.
When data variation type is that data are deleted, it is interim that data are deleted into corresponding second variation record write-in second
Table.Correspondingly, data are deleted into corresponding second variation record the second interim table of write-in, it can be with specifically: first more according to data
New corresponding second variation, which records, generates corresponding 2nd Sql instruction, and the second variation is recorded batch further according to the 2nd Sql instruction
It is inserted into the second interim table.
It is easily understood that either the first Sql is instructed or the 2nd Sql instruction, for according to variation record generation
Corresponding Sql instruction is inserted into interim table for realizing that will change record batch.Herein for two Sql instruction particular content and
Format is not construed as limiting.
When existing simultaneously data update and data are deleted, S102 is specifically as follows:
Data are updated into corresponding first variation record the first interim table of write-in, data are deleted into corresponding second variation and are remembered
Record the second interim table of write-in.
This step is specifically described in citing below, it is assumed that there are source table (i.e. the first data of a real-time update
Table) A, the data in the Table A of source have occurred insertion, update and delete variation.It first needs the delta data in Table A being synchronized to purpose
In table B.
During synchronizing, it will be carried out in three steps according to the type of delta data:
Purpose table or interim table is written in each record by the first step:
1. data are inserted into, insertion is recorded into batch write-in purpose table B (dst_test).
2. data update, will more new record namely first variation record, batch be written the first interim table C (dst_test_
update)。
3. data are deleted, deletion record namely the second variation are recorded, the second interim table D (dst_test_ is written in batch
delete)。
It can be seen that, although being all the interim table of write-in, the two should when existing simultaneously data update and data are deleted
Corresponding different interim table.
Second step, according to the more new record in the first interim table C, execute Sql and update respective record in purpose table B:
It can be such that specifically, this step executes Sql operating process
update dst_test B
Inner join dst_test_update C on B.ID=C.ID
Set B.TFSJ=C.TFSJ；
truncate dst_test_update；
Therein, it can be seen that in the presence of the process for merging the interim table C of purpose table B and first.Pass through purpose table B and first
Interim table C is associated with by identity field (ID), by the first interim table C field value (TFSJ) write-in purpose table B corresponding field (TFSJ).
Second step, according to the deletion record in the second interim table D, execute Sql and delete respective record in purpose table B:
It can be such that specifically, this step executes Sql operating process
delete from dst_test
where ID in(select D.ID from dst_test_delete D)；
truncate dst_test_delete；
Therein, it can be seen that in the presence of the process for merging the interim table D of purpose table B and second.It deletes in purpose table B and includes
The record of second interim table D identity field value (ID).
It should be pointed out that the application is only applied to the specific implementation process of one of database by the present embodiment,
Under the premise of not departing from the application principle, there can also be several improvement to the present embodiment.For example, second step and third step are held
Row reversed order, or the first step is put to last etc., in short, the first step, second step, third step are on the sequencing of step
It does not limit, either implementing in which way should all be within the scope of protection of this application.
The data synchronous system between a kind of database provided by the embodiments of the present application is introduced below, is described below
Data synchronous system can correspond to each other reference with above-described method of data synchronization.
Data synchronous system structural schematic diagram of the Fig. 2 between a kind of database provided by the embodiment of the present application, this Shen
A kind of data synchronous system between database is please also provided, comprising:
Judgment module 100, for that ought detect the first tables of data, there are delta datas, and need the delta data is same
When step to the second tables of data, judge whether data variation type is that data update or data are deleted；
Writing module 200, for interim table to be written in the variation record of the delta data；
It is synchronous to complete data for merging the interim table with second tables of data for synchronization module 300.
Based on data synchronous system described above, as preferred embodiment, wherein interim table may include first facing
When table and the second interim table.
Further, when data variation type is that data update, writing module 200 can be used for updating data corresponding
First variation record the first interim table of write-in；
Further, when data variation type is that data are deleted, writing module 200 can be used for deleting data corresponding
Second variation record the second interim table of write-in；
Further, when data variation type is that data update and data are deleted, writing module 200 can be used for data
Corresponding first variation record the first interim table of write-in is updated, it is interim that data are deleted corresponding second variation record write-in second
Table.
Based on the above embodiment, as preferred embodiment, writing module 200 can update pair particularly for by data
The the first variation record batch answered is inserted into the module of the first interim table.
Based on the above embodiment, as preferred embodiment, writing module 200 can update pair particularly for by data
The the second variation record batch answered is inserted into the module of the second interim table.
Based on the above embodiment, as preferred embodiment, the data synchronous system can also include:
Temp table generation module for generating the interim table, and is associated with the interim table and second tables of data.
Based on the above embodiment, as preferred embodiment, the synchronization module 300 is specifically as follows for according to
The delta data batch is incorporated to second tables of data by the variation record in interim table using Sql, completes data
Synchronous module.
Based on the above embodiment, as preferred embodiment, the data synchronous system can also include:
Message generating module, for generating the synchronous success message of data.
Present invention also provides a kind of servers, referring to Fig. 3, a kind of structure of server provided by the embodiments of the present application
Figure, as shown in figure 3, may include processor 11 and memory 12.Referring to fig. 4, which can also include multimedia group
Part 13, one or more of input/output (I/O) interface 14 and communication component 15.
Wherein, processor 11 is used to control the integrated operation of the server 400, to complete in above-mentioned method of data synchronization
All or part of the steps.Memory 12 is used to store various types of data to support the operation in the server 400, these
Data for example may include the instruction of any application or method for operating on the server 400, and apply journey
The relevant data of sequence, such as contact data, the message of transmitting-receiving, picture, audio, video etc..The memory 12 can be by appointing
The volatibility or non-volatile memory device or their combination of what type are realized, such as static random access memory
(Static Random Access Memory, abbreviation SRAM), electrically erasable programmable read-only memory (Electrically
Erasable Programmable Read-Only Memory, abbreviation EEPROM), Erasable Programmable Read Only Memory EPROM
(Erasable Programmable Read-Only Memory, abbreviation EPROM), programmable read only memory
(Programmable Read-Only Memory, abbreviation PROM), and read-only memory (Read-Only Memory, referred to as
ROM), magnetic memory, flash memory, disk or CD.Multimedia component 13 may include screen and audio component.Wherein shield
Curtain for example can be touch screen, and audio component is used for output and/or input audio signal.For example, audio component may include one
A microphone, microphone is for receiving external audio signal.The received audio signal can be further stored in memory
It 12 or is sent by communication component 15.Audio component further includes at least one loudspeaker, is used for output audio signal.I/O interface
14 provide interface between processor 11 and other interface modules, other above-mentioned interface modules can be keyboard, mouse, button
Deng.These buttons can be virtual push button or entity button.Communication component 15 is used between the server 400 and other equipment
Carry out wired or wireless communication.Wireless communication, such as Wi-Fi, bluetooth, near-field communication (Near Field Communication,
Abbreviation NFC), 2G, 3G or 4G or they one or more of combination, therefore the corresponding communication component 15 may include:
Wi-Fi module, bluetooth module, NFC module.
In one exemplary embodiment, server 400 can be by one or more application specific integrated circuit
(Application Specific Integrated Circuit, abbreviation ASIC), digital signal processor (Digital
Signal Processor, abbreviation DSP), digital signal processing appts (Digital Signal Processing Device,
Abbreviation DSPD), programmable logic device (Programmable Logic Device, abbreviation PLD), field programmable gate array
(Field Programmable Gate Array, abbreviation FPGA), controller, microcontroller, microprocessor or other electronics member
Part is realized, for executing above-mentioned method of data synchronization.
In a further exemplary embodiment, a kind of computer readable storage medium including program instruction is additionally provided, it should
The step of above-mentioned method of data synchronization is realized when program instruction is executed by processor.For example, the computer readable storage medium can
Think the above-mentioned memory 12 including program instruction, above procedure instruction can be executed by the processor 11 of server 400 to complete
Above-mentioned method of data synchronization.
Each embodiment is described in a progressive manner in specification, the highlights of each of the examples are with other realities
The difference of example is applied, the same or similar parts in each embodiment may refer to each other.For embodiment provide system and
Speech, since it is corresponding with the method that embodiment provides, so being described relatively simple, related place is referring to method part illustration
?.
Specific examples are used herein to illustrate the principle and implementation manner of the present application, and above embodiments are said
It is bright to be merely used to help understand the present processes and its core concept.It should be pointed out that for the ordinary skill of the art
For personnel, under the premise of not departing from the application principle, can also to the application, some improvement and modification can also be carried out, these improvement
It is also fallen into the protection scope of the claim of this application with modification.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also including other elements that are not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or apparatus that includes the element.
Test result comparison
Oracle Data source table is respectively configured to the synchronous of Gbase tables of data with the technical program according to prior art scheme
Task, source table, purpose table include 1,000,000 datas, and are sequentially inserted into the table of source, update, delete 1000 datas, are matched
It sets scheme and test result is as follows:
One, incremental data importing is carried out using precompile SQL statement after prior art capture incremental data, update every time,
Data are deleted to need first to be inquired.The insertion of 1000 datas synchronizes time-consuming 5 seconds, and the update of 1000 datas synchronizes 8 point 32 time-consuming
Second, 1000 datas delete time-consuming 8 seconds, add up time-consuming 8 points 45 seconds.
Two, different disposal is done according to incremental data type after the technical program capture incremental data, mesh is written in insertion data
Table, more new data be written the first interim table, delete data be written the second interim table.The insertion of 1000 datas is synchronized to purpose table
5 seconds time-consuming, the update of 1000 datas is synchronized to first interim table time-consuming 6 seconds, and the deletion of 1000 datas is synchronized to the second interim table
It is 5 seconds time-consuming, it executes interim table and merges Sql time-consuming 0.71 second, add up 16.71 seconds time-consuming.
Claims (10)
1. a kind of method of data synchronization characterized by comprising
When detecting that the first tables of data there are delta data, and when needing the delta data being synchronized to the second tables of data, sentences
Whether disconnected data variation type is that data update and/or data are deleted；
If so, interim table is written in the variation record of the delta data；Wherein, the interim table and second tables of data
Table structure is identical；
The interim table is merged with second tables of data, in second tables of data according to the variation recording synchronism
After delta data, it is synchronous to complete data.
2. method of data synchronization according to claim 1, which is characterized in that the interim table includes the first interim table and the
Two interim tables；
When the data variation type is that data update, data are updated into corresponding first variation record write-in described first and are faced
When table；
When the data variation type is that data are deleted, data are deleted into corresponding second variation record write-in described second and are faced
When table；
When the data variation type is that data update and data are deleted, data are updated into corresponding first variation record write-in
Data are deleted corresponding second variation record and the second interim table are written by the first interim table.
3. method of data synchronization according to claim 2, which is characterized in that described that data are updated corresponding first variation
The first interim table is written in record, comprising:
Data are updated into corresponding first variation record batch and are inserted into the described first interim table.
4. method of data synchronization according to claim 2, which is characterized in that described that data are deleted corresponding second variation
The second interim table is written in record, comprising:
Data are updated into corresponding second variation record batch and are inserted into the described second interim table.
5. method of data synchronization according to claim 1-4, which is characterized in that by the variation of the delta data
Before interim table is written in record, further includes:
The interim table is generated, and is associated with the interim table and second tables of data.
6. method of data synchronization according to claim 5, which is characterized in that by the interim table and second tables of data
Merge, completion data, which synchronize, includes:
The delta data batch is incorporated to second data using Sql according to the variation record in the interim table
It is synchronous to complete data for table.
7. method of data synchronization according to claim 1, which is characterized in that by the interim table and second tables of data
Merge, after completion data are synchronous further include:
Generate the synchronous success message of data.
8. the data synchronous system between a kind of database characterized by comprising
Judgment module, for when detecting that the first tables of data there are delta data, and needs for the delta data to be synchronized to the
When two tables of data, judge whether data variation type is that data update or data are deleted；
Writing module, for interim table to be written in the variation record of the delta data；Wherein, the interim table and described second
Tables of data table structure is identical；
It is synchronous to complete data for merging the interim table with second tables of data for synchronization module.
9. a kind of server, which is characterized in that including memory and processor, have computer program, institute in the memory
It states when processor calls the computer program in the memory and realizes such as the described in any item data sides of synchronization claim 1-7
Method.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
Such as claim 1-7 described in any item method of data synchronization are realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910063590.5A CN109739936A (en) | 2019-01-23 | 2019-01-23 | Method of data synchronization, system, server and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910063590.5A CN109739936A (en) | 2019-01-23 | 2019-01-23 | Method of data synchronization, system, server and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109739936A true CN109739936A (en) | 2019-05-10 |
Family
ID=66365766
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910063590.5A Pending CN109739936A (en) | 2019-01-23 | 2019-01-23 | Method of data synchronization, system, server and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109739936A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110333970A (en) * | 2019-07-10 | 2019-10-15 | 中国民航信息网络股份有限公司 | A kind of data reconstruction method and device |
| CN110442585A (en) * | 2019-06-21 | 2019-11-12 | 中国平安财产保险股份有限公司 | Data-updating method, data update apparatus, computer equipment and storage medium |
| CN110659281A (en) * | 2019-08-14 | 2020-01-07 | 中国平安财产保险股份有限公司 | Hive-based data processing method and device, computer equipment and storage medium |
| CN111259082A (en) * | 2020-02-11 | 2020-06-09 | 车配齐(珠海横琴)科技有限公司 | Method for realizing full data synchronization in big data environment |
| CN111752910A (en) * | 2020-06-24 | 2020-10-09 | 上海微盟企业发展有限公司 | Data synchronization method, system and related device for heterogeneous platform |
| CN112231331A (en) * | 2020-10-23 | 2021-01-15 | 金蝶云科技有限公司 | Inventory data synchronization method and inventory data synchronization module |
| CN112328696A (en) * | 2020-11-17 | 2021-02-05 | 北京金和网络股份有限公司 | Method and device for synchronizing data in MySQL database to SQL Server database |
| CN113704254A (en) * | 2021-07-30 | 2021-11-26 | 北京奇艺世纪科技有限公司 | Database service processing method and device, electronic equipment and storage medium |
| CN114297216A (en) * | 2021-12-30 | 2022-04-08 | 北京金堤科技有限公司 | Data synchronization method and device, computer storage medium and electronic equipment |
| WO2022083266A1 (en) * | 2020-10-19 | 2022-04-28 | 中兴通讯股份有限公司 | Data table synchronization method and apparatus, data exchange device, and storage medium |
| CN110442585B (en) * | 2019-06-21 | 2024-04-30 | 中国平安财产保险股份有限公司 | Data updating method, data updating device, computer equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060063358A (en) * | 2004-12-07 | 2006-06-12 | 에스케이 텔레콤주식회사 | Data transmission service system based on asynchronous method in wireless portal platform and method thereof |
| CN102495910A (en) * | 2011-12-28 | 2012-06-13 | 畅捷通信息技术股份有限公司 | Device and method for data timing synchronization of heterogeneous system |
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN103605776A (en) * | 2013-11-28 | 2014-02-26 | 北京国双科技有限公司 | Method and device for processing data of information database |
| CN105701218A (en) * | 2016-01-14 | 2016-06-22 | 四川长虹电器股份有限公司 | Method for realizing data synchronism of different terminals on database |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
-
2019
- 2019-01-23 CN CN201910063590.5A patent/CN109739936A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20060063358A (en) * | 2004-12-07 | 2006-06-12 | 에스케이 텔레콤주식회사 | Data transmission service system based on asynchronous method in wireless portal platform and method thereof |
| CN102495910A (en) * | 2011-12-28 | 2012-06-13 | 畅捷通信息技术股份有限公司 | Device and method for data timing synchronization of heterogeneous system |
| CN102982126A (en) * | 2012-11-14 | 2013-03-20 | 浙江星汉信息技术有限公司 | Method and system of real-time delivery of database table data |
| CN103605776A (en) * | 2013-11-28 | 2014-02-26 | 北京国双科技有限公司 | Method and device for processing data of information database |
| CN105701218A (en) * | 2016-01-14 | 2016-06-22 | 四川长虹电器股份有限公司 | Method for realizing data synchronism of different terminals on database |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
Non-Patent Citations (1)
| Title |
|---|
| 彭磊: "大数据量Excel数据导入系统的设计与实现", 《现代电子技术》 * |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110442585A (en) * | 2019-06-21 | 2019-11-12 | 中国平安财产保险股份有限公司 | Data-updating method, data update apparatus, computer equipment and storage medium |
| CN110442585B (en) * | 2019-06-21 | 2024-04-30 | 中国平安财产保险股份有限公司 | Data updating method, data updating device, computer equipment and storage medium |
| CN110333970B (en) * | 2019-07-10 | 2021-09-07 | 中国民航信息网络股份有限公司 | Data recovery method and device |
| CN110333970A (en) * | 2019-07-10 | 2019-10-15 | 中国民航信息网络股份有限公司 | A kind of data reconstruction method and device |
| CN110659281A (en) * | 2019-08-14 | 2020-01-07 | 中国平安财产保险股份有限公司 | Hive-based data processing method and device, computer equipment and storage medium |
| CN110659281B (en) * | 2019-08-14 | 2023-09-12 | 中国平安财产保险股份有限公司 | Hive-based data processing method, hive-based data processing device, computer equipment and storage medium |
| CN111259082B (en) * | 2020-02-11 | 2023-07-21 | 深圳市六因科技有限公司 | Method for realizing full data synchronization in big data environment |
| CN111259082A (en) * | 2020-02-11 | 2020-06-09 | 车配齐(珠海横琴)科技有限公司 | Method for realizing full data synchronization in big data environment |
| CN111752910A (en) * | 2020-06-24 | 2020-10-09 | 上海微盟企业发展有限公司 | Data synchronization method, system and related device for heterogeneous platform |
| WO2022083266A1 (en) * | 2020-10-19 | 2022-04-28 | 中兴通讯股份有限公司 | Data table synchronization method and apparatus, data exchange device, and storage medium |
| CN112231331A (en) * | 2020-10-23 | 2021-01-15 | 金蝶云科技有限公司 | Inventory data synchronization method and inventory data synchronization module |
| CN112328696A (en) * | 2020-11-17 | 2021-02-05 | 北京金和网络股份有限公司 | Method and device for synchronizing data in MySQL database to SQL Server database |
| CN113704254A (en) * | 2021-07-30 | 2021-11-26 | 北京奇艺世纪科技有限公司 | Database service processing method and device, electronic equipment and storage medium |
| CN113704254B (en) * | 2021-07-30 | 2023-06-16 | 北京奇艺世纪科技有限公司 | Method and device for processing database business, electronic equipment and storage medium |
| CN114297216B (en) * | 2021-12-30 | 2022-09-02 | 北京金堤科技有限公司 | Data synchronization method and device, computer storage medium and electronic equipment |
| CN114297216A (en) * | 2021-12-30 | 2022-04-08 | 北京金堤科技有限公司 | Data synchronization method and device, computer storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109739936A (en) | Method of data synchronization, system, server and computer readable storage medium | |
| US8538930B2 (en) | Method and system for database recovery | |
| CN109241175A (en) | Method of data synchronization, device, storage medium and electronic equipment | |
| CN102222071B (en) | Method, device and system for data synchronous processing | |
| CN104965879A (en) | Method and device for altering table structure of data table | |
| CN111708841B (en) | Database data synchronization method and device | |
| CN108205560B (en) | Data synchronization method and device | |
| US20100042638A1 (en) | Apparatus, method, and computer program product for synchronizing data sources | |
| CN104834749A (en) | File synchronization method and synchronization system for mobile terminal and PC terminal | |
| CN111414424A (en) | Method, system, medium and device for automatically synchronizing redis of configuration data | |
| CN109542329B (en) | Log writing method, storage management equipment and system | |
| CN110874358A (en) | Multi-attribute column storage and retrieval method and device and electronic equipment | |
| CN113157670B (en) | Data migration method and system of database | |
| CN109271442A (en) | Database synchronization restoration methods, device, computer readable storage medium and electronic equipment | |
| CN110647421B (en) | Database processing method, device and system and electronic equipment | |
| CN111078418A (en) | Operation synchronization method and device, electronic equipment and computer readable storage medium | |
| CN109614442A (en) | Data synchronous data listing maintenance, device, storage medium and electronic equipment | |
| CN111263195B (en) | Barrage processing method and device, server equipment and storage medium | |
| CN110795318B (en) | Data processing method and device and electronic equipment | |
| CN107491363A (en) | A kind of Snapshot Method and device of the storage volume based on linux kernel | |
| CN110865897B (en) | Conflict detection method and device and electronic equipment | |
| CN113742415A (en) | Incremental data synchronization method and device and electronic equipment | |
| CN111026810B (en) | Data synchronization method, device and storage medium | |
| CN110569231A (en) | Data migration method, device, equipment and medium | |
| CN112988860A (en) | Data acceleration processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |