CN109189852A - A kind of method that data are synchronous and the device synchronous for data - Google Patents

A kind of method that data are synchronous and the device synchronous for data Download PDF

Info

Publication number
CN109189852A
CN109189852A CN201810860116.0A CN201810860116A CN109189852A CN 109189852 A CN109189852 A CN 109189852A CN 201810860116 A CN201810860116 A CN 201810860116A CN 109189852 A CN109189852 A CN 109189852A
Authority
CN
China
Prior art keywords
log
data
definition information
database
offline dictionary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810860116.0A
Other languages
Chinese (zh)
Other versions
CN109189852B (en
Inventor
付铨
孙峰
陈艳
杨春
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Dream Database Co ltd
Original Assignee
Wuhan Dameng Database Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Dameng Database Co Ltd filed Critical Wuhan Dameng Database Co Ltd
Priority to CN201810860116.0A priority Critical patent/CN109189852B/en
Publication of CN109189852A publication Critical patent/CN109189852A/en
Application granted granted Critical
Publication of CN109189852B publication Critical patent/CN109189852B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention relates to database synchronization technical fields, provide a kind of method that data are synchronous and the device synchronous for data, and the synchronous method of the data includes establishing the offline dictionary comprising object definition information;Obtain the operation log in source database log file;Log serial number according to operation log obtains corresponding object definition information from offline dictionary, and is parsed according to the object definition information got to operation log;It will carry out the data that log parses and be sent to purpose client database.The synchronous method of data of the present invention can be by accessing offline dictionary, it establishes and the complementary parsing of the operation log in the prior art only comprising local key message, the case where not only ensure that the correctness of the definition information of object in operation log, while also avoiding real time access database and influencing the performance of database.

Description

A kind of method that data are synchronous and the device synchronous for data
[technical field]
It is same more particularly to a kind of synchronous method of data and for data the present invention relates to database synchronization technical field The device of step.
[background technique]
Database synchronization technology is now widely used for all kinds of emerging products such as distributed system, big data, cloud database In, the immense pressure that the access of big data quantity high concurrent brings system can be effectively relieved by the redundancy scheme of database.Pass through The transaction journal for extracting source database, obtains the data manipulation of source database, including INSERT, DELETE, UPDATE, then leads to It crosses network and is sent to purpose client database, destination reverts to original SQL, and (Structured Query Language, writes a Chinese character in simplified form SQL) sentence executes on purpose database, is a kind of common database synchronization technology.The database synchronization technology have pair The performance and data mode influences of source database are small, and very flexible mode can be used and configure various topological structures, support across The advantages that operating system and database copy of platform of isomery.
Currently, the journal file in database represents operation object using object identity, without will record each object Corresponding object definition information.During using the analysis transaction journal of log analysis technology, need through object identity reality When queried access database obtain accurate object definition information, correct log parsing work could be completed.
But it when by connection database to obtain object definition information, can have the following problems: on the one hand, due to It only will record existing object in Database Systems dictionary and define information (current time newest definition information), therefore, log point Real-time query database possibly can not get the correct definition information of the target operation object of operation log recording during analysis. For example, before object definition modification occurs for DML (Data Manipulation Language, the DML) operation of some object, Then record determining for the object recorded in the definition information and current time database of the corresponding object of operation log of DML operation Adopted information be it is different, therefore, at this time by access database system dictionary in object definition, operation log can not be obtained The correct definition of middle object.On the other hand, real-time query database during log analysis, has an impact to database performance. Furthermore it deposits to be easy to cause when abnormal when database and can not normally obtain object definition information.
Therefore, traditional log analysis technology is in the way of real-time query database acquisition object definition information, not only It not can guarantee the correctness of object definition, and then influence the synchronous accuracy of data, while affecting the performance of database.
In consideration of it, overcoming defect present in the prior art is the art urgent problem to be solved.
[summary of the invention]
The technical problem to be solved by the present invention is to can not obtain behaviour when carrying out data synchronization by the way of log analysis The correct definition information for making the corresponding operation object of log leads to data synchronous the problem of malfunctioning.
The embodiment of the present invention adopts the following technical scheme that
In a first aspect, the present invention provides a kind of method that data are synchronous, the synchronous method of the data includes: that foundation includes The offline dictionary of object definition information;Obtain the operation log in source database log file;According to the operation log Log serial number obtains corresponding object definition information from the offline dictionary, and according to the object definition information pair got The operation log is parsed;It will carry out the data that log parses and be sent to purpose client database.
Preferably, described to establish after the offline dictionary comprising object definition information further include: to obtain purpose client database Data synchronous point, and determine origin log sequence number according to the data synchronous point;The acquisition source database journal text Operation log in part includes: to obtain log serial number in source database log file to be not less than the origin log sequence number Operation log.
Preferably, the offline dictionary of the foundation comprising object definition information includes: to obtain source Database Systems dictionary In object definition information and the corresponding current log sequence number of current time source database;By the object definition information and The current log sequence number is integrated into the offline dictionary file of first version, and the offline dictionary file of the first version is stored In local disk;The data manipulation for parsing source database, updates the offline dictionary file of the corresponding object of the data manipulation.
Preferably, the object definition information and current time source database obtained in source Database Systems dictionary Corresponding current log sequence number includes: based on structured query language query source client database system dictionary;Obtain with it is described The definition information and the corresponding current log sequence of current time source database for the object that structured query language matches Number;Wherein, defining information includes the one or more in schema name, object name and district location.
Preferably, the data manipulation is creation object;The data manipulation of the parsing source database, updates the number It include: to judge determine whether the object of the source database creation is needs according to the offline dictionary file for operating corresponding object Synchronous object;If needing synchronous object, definition information and the creation Object Operations for obtaining the object of creation are corresponding Log serial number;The definition information of the object of creation and corresponding log serial number are integrated into the offline dictionary text of first version Part, and the offline dictionary file of the first version is stored in local disk.
Preferably, the data manipulation is modification object;The data manipulation of the parsing source database, updates the number It include: the definition information and modification Object Operations for obtaining modified object according to the offline dictionary file for operating corresponding object Corresponding log serial number;By the definition information of modified object and corresponding log serial number be integrated into next version from Line dictionary file.
Preferably, the data manipulation is to delete object;The data manipulation of the parsing source database, updates the number It include: to obtain to delete the corresponding log serial number of Object Operations according to the offline dictionary file for operating corresponding object;Judgement determines Whether the corresponding log serial number of the deletion Object Operations is less than the origin log sequence number;If being less than, by the quilt The corresponding offline dictionary file of object is deleted to delete from local disk.
Preferably, the log serial number according to the operation log obtains corresponding object from the offline dictionary Information is defined, and carrying out parsing to the operation log according to the object definition information got includes: the parsing operation day Will is to obtain target operation object;Obtain the offline dictionary file of all versions of the definition information comprising the target operation object Corresponding log serial number;The log serial number for comparing the operation log is corresponding with all offline dictionary files of version Log serial number;Based on the offline dictionary file for determining respective version greater than nearest principle, to obtain the target operation object Definition information.
Preferably, the object is one or more of table, index and view.
Second aspect, the present invention provide a kind of device synchronous for data, including at least one processor;And with The memory of at least one processor communication connection;Wherein, the memory be stored with can by it is described at least one processing The instruction that device executes, described instruction are arranged to carry out the synchronous method of data described in first aspect by program.
The third aspect, the present invention also provides a kind of nonvolatile computer storage media, the computer storage medium Computer executable instructions are stored with, which is executed by one or more processors, for completing first The synchronous method of data described in aspect.
Compared with prior art, the beneficial effect of the embodiment of the present invention is: being counted by the way of log analysis When according to synchronizing, the offline dictionary comprising object definition information is initially set up, the accessible offline dictionary when carrying out log analysis, It establishes and is parsed with the complementary of the operation log in the prior art only comprising local key message, it is right in operation log not only to ensure that The correctness of the definition information of elephant, while the case where also avoid real time access database and influence the performance of database.
[Detailed description of the invention]
In order to illustrate the technical solution of the embodiments of the present invention more clearly, will make below to required in the embodiment of the present invention Attached drawing is briefly described.It should be evident that drawings described below is only some embodiments of the present invention, for For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other Attached drawing.
Fig. 1 is a kind of flow diagram of the synchronous method of data provided in an embodiment of the present invention;
Fig. 2 is the idiographic flow schematic diagram of step 10 in Fig. 1;
Fig. 3 is the idiographic flow schematic diagram of an embodiment of step 103 in Fig. 2;
Fig. 4 is the idiographic flow schematic diagram of another embodiment of step 103 in Fig. 2;
Fig. 5 is the idiographic flow schematic diagram of the another embodiment of step 103 in Fig. 2;
Fig. 6 is a kind of structural schematic diagram of device synchronous for data provided in an embodiment of the present invention.
[specific embodiment]
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and It is not used in the restriction present invention.
Embodiment 1:
In practical application scene, when synchronous using log analysis progress data, it is necessary to be believed using correct object definition Breath, object definition information is excessively new or excessively old, all may cause analysis or timing error.In the present embodiment, by source data Object definition information in the system dictionary of library is stored in local, during carrying out log analysis, accesses local offline dictionary , it is not necessarily to real time access database, can not only reduce the influence to database performance, while can also guarantee to obtain correct Object definition information, and then guarantee that data are synchronous or the accuracy of log analysis.Be specifically described below using offline dictionary into The synchronous method of row data.
Step 10: establishing the offline dictionary comprising object definition information.
In the present embodiment, source database is first depending on actual demand and establishes the offline word comprising object definition information Allusion quotation.
Wherein, source database include SQLSERVER database, MYSQL database, DB2 database, DM6 database, Any one of DM7 database, POSTGRESQL database or ORACLE database, the type of database does not do specific limit herein It is fixed, it can be selected according to actual conditions.
Wherein, object includes one of table, index and view or a variety of.
Specifically, what table was made of row and column, column are made of similar information, and each column is also known as a field, often The title of column is known as field name, and row includes several columns item of information.Data line is known as one or one record, it, which is expressed, one Determine the information combination of meaning.One table can be made of one or more record, and unwritten table is known as empty table.
Index is the sequence set up according to specified data Ku Biaolie, and index provides the way of quickly access data Diameter, and the data of table can be supervised, the data in column for keeping its index pointed do not repeat, such as clustered index.
View seems that same table seems the same, and view is field and data item with one group of name, but view its It is in fact a virtual table, not physical presence in the database.View is generated by inquiry database table, it limits use The data that family can be seen and modify.View can be used to control access of the user to data, and can simplify the display of data, i.e., logical It crosses view and only shows those data informations needed.
Below referring to Fig. 2, illustrating the method for establishing offline dictionary.
Step 101: obtaining the object definition information and current time source database pair in source Database Systems dictionary The current log sequence number answered.
In the present embodiment, (SQL) source Database Systems dictionary, acquisition and structure are inquired based on structured query language Change the definition information for the object that query language matches.In practical application scene, can according to actual demand design SQL statement with Obtain the definition information of corresponding object.Wherein, the definition information of object includes one in schema name, object name and district location Item is multinomial.In an alternate embodiment of the invention, wherein the definition information of object includes schema name, object name and district location. For example, the definition information of object includes the district location of schema name, table name, column name and table when object is table.
In practical application scene, when Database Systems type difference, the mode for obtaining object definition information is also poor It is different.To be illustrated so that Database Systems are ORACLE database, object is table as an example obtain object herein convenient for illustrating Define the mode of information.
Firstly, obtaining the initialization information of table using select sentence, form sentence and where sentence.For example, in reality The initialization information of table: select sc.obj#, sc.intcol#, sc.name is obtained in application scenarios in the following way, Sc.type#, sc.charset form (select dataobj#from sys.obj $ where obj#=(select Lobj#from sys.lob $ sl where sl.obj#=sc.obj#and sl.intcol#=sc.intcol#)) lobid, Sc.length, sc.precision#, sc.scale, sc.null $, sc.default $, sc.property, (select Toid from sys.coltype $ where obj#=sc.obj#and intcol#=sc.intcol#), sc.segcol#, sc.col#from sys.col$sc WHERE SC.OBJ#IN(select tab.obj#from sys.obj$tab left Join sys.user $ sch on (tab.owner#=sch.user#) where tab.type#=2and sch.type#= 1and tab.obj#not in (select distinct obj#from sys.col $ where type#not in (1,2,8, 9,12,23,24,25,58,69,96,100,101,112,113,121,122,123,180,181,182,183,208,231))) order by sc.segcol#。
Then, the column information of table is obtained using select sentence, form sentence and where sentence.For example, in practical application The column information of table: select sc.obj#, sc.intcol#, sc.name, sc.type# is obtained in scene in the following way, Sc.charsetform, (select dataobj#from sys.obj $ where obj#=(select lobj#from Sys.lob $ sl where sl.obj#=sc.obj#and sl.intcol#=sc.intcol#)) lobid, sc.length, Sc.precision#, sc.scale, sc.null $, sc.default $, sc.property, (select toid from Sys.coltype $ where obj#=sc.obj#and intcol#=sc.intcol#), sc.segcol#, sc.col# from sys.col$sc WHERE SC.OBJ#IN(select tab.obj#from sys.obj$tab left join Sys.user $ sch on (tab.owner#=sch.user#) where tab.type#=2and sch.type#=1and Tab.obj#not in (select distinct obj#from sys.col $ where type#not in (1,2,8,9,12, 23,24,25,58,69,96,100,101,112,113,121,122,123,180,181,182,183,208,231))) order by sc.segcol#。
Finally, obtaining the partition information of table using select sentence, form sentence and where sentence.For example, actually answering With the partition information for obtaining table in scene in the following way: select tp.obj#, tp.dataobj#, tp.bo#, Tab.subname from sys.tabpart $ tp, sys.obj $ tab where tp.obj#=tab.obj#union all Select a.obj#, a.dataobj#, b.bo#, c.subname from sys.tabsubpart $ a, sys.tabcompart $ b, sys.obj $ c where a.pobj#=b.obj#and a.obj#=c.obj#union all select Tcp.obj#, tcp.dataobj#, tcp.bo#, tab.subname from sys.tabcompart $ tcp, sys.obj $ tab Where tcp.obj#=tab.obj#
// obtain partition information
Select obj#, parttype, mod (SPARE2,256) from sys.partobj $
// partition table blob
Select a.dataobj#, b.tabfragobj#, a.name from sys.obj $ a left join Sys.lobfrag $ b on (a.obj#=b.fragobj#) where (a.type#=40or a.type#=41).
Above-mentioned only illustrating table according to specific instance interpretation defines information acquiring pattern, in practical application scene In, corresponding SQL statement can be write according to the type of database, and then acquisition table defines information.
It include at least one offline dictionary file in the present embodiment, offline dictionary, each offline dictionary file corresponding one The definition information of a object, that is, may include the definition information of at least one object in offline dictionary.Due in practical application field Jing Zhong, the definition information of object are possible to be modified, in order to distinguish some object in the definition information of different moments, in this implementation In example, each offline dictionary file has the offline dictionary file of at least one version.In order to distinguish the same object difference version This offline dictionary file, in the present embodiment, the version number of the offline dictionary file of configuration log Serial No..
Specifically, obtaining the corresponding current log sequence of current time source database when obtaining object definition information Row number, wherein current log sequence number can be the maximum logged sequence in the journal file of current time source database Number.Wherein, log serial number is the numerical value safeguarded automatically by Database Systems, has automatic be incremented by and globally unique characteristic.
In practical application scene, since there are a large amount of objects in Database Systems, operated to each object Shi Junhui is corresponding to generate operation log, and system can automatically generate log serial number to distinguish different operations at the same time.Log Sequence number is used to represent the physics affairs generated inside Database Systems, log text of the log serial number in Database Systems There is global uniqueness, to distinguish different physics affairs in part.Currently, most of data base management systems use LSN (Log Sequence number, abbreviation LSN) the physics affairs generated inside Database Systems are represented, for example, SQLSERVER, The data base management systems such as MYSQL, DB2, DM6, DM7.But, in ORACLE database, using SCN (System Change Number, abbreviation SCN) as the mark for generating a physics affairs, and the LSN in ORACLE is only used as one that log switches Serial number then can use SCN as log serial number in ORACLE database.
In addition, the mode for obtaining log serial number can also have differences accordingly when Database Systems type difference.Example Such as, when Database Systems are ORACLE database, SCN can be inquired, specifically as log serial number by SQL statement Select dbms_flashback.get_system_change_number from dual order can be used to obtain SCN value (log serial number).When Database Systems are DM7 database, log-sequence numbers LSN can pass through SQL language as log serial number Sentence is inquired, and select cur_lsn from sys.v $ rlog order can be used specifically to obtain LSN value (logged sequence Number).When Database Systems are DM6 database, log-sequence numbers LSN can be looked into as log serial number by SQL statement It askes, SELECT GET_LAST_LSN (' database name ') order can be used specifically to obtain LSN value (log serial number).Work as number According to library system be MYSQL database when, log-sequence numbers LSN can be inquired, specifically as log serial number by SQL statement Show binary logs order can be used to obtain LSN value (log serial number).When Database Systems are POSTGRESQL number When according to library, log-sequence numbers LSN can be inquired as log serial number by SQL statement, and SELECT cast specifically can be used (pg_current_xlog_insert_location ()-cast (' 0/0'as pg_lsn) as bigint) order to obtain LSN value (log serial number).When Database Systems are SQLSERVER database, log-sequence numbers LSN as log serial number, It can be inquired by SQL statement, select sys.fn_cdc_get_max_lsn () order can be used specifically to obtain LSN It is worth (log serial number).When Database Systems are DB2 database, log-sequence numbers LSN can pass through Api as log serial number Interface is obtained, and db2ReadLog function can be used specifically to obtain LSN value (log serial number).
The aforementioned acquisition modes for only enumerating simple declaration log serial number, can specifically use according to practical application scene SQL statement or Api interface obtain log serial number, and details are not described herein.
Step 102: object definition information and current log sequence number being integrated into the offline dictionary file of first version, and will The offline dictionary file of first version is stored in local disk.
In the present embodiment, after the definition information and current log sequence number for obtaining object, object is corresponding It defines information and is integrated into the offline dictionary file of first version, and offline dictionary file is stored in local disk.Wherein, the first edition The benchmark that the definition information of object can be used as object in this offline dictionary file defines information.It is modified in object definition information When, it can be by after modified object definition information adding to the offline dictionary file of first version, and configure object modification behaviour Make the version number that corresponding log serial number is the version, to distinguish.
Specifically, the definition information of each object can be integrated into offline dictionary file with txt text formatting.Due to The number of object be it is multiple, correspondingly the number of the offline dictionary file in offline dictionary is also multiple, different in order to distinguish Offline dictionary file, the present embodiment are also that the corresponding offline dictionary file of each object distributes unique document identification number, for After the corresponding offline dictionary file of each object distributes unique document identification number, offline dictionary file is stored in this earth magnetism Disk.Wherein, document identification number is that Database Systems are distributed automatically, has global uniqueness.
After definition information that object is corresponding is integrated into offline dictionary file, configuring the log serial number is offline dictionary The offline dictionary file for having configured version number is finally stored in local disk again by the version number of file.
Step 103: the data manipulation of parsing source database updates the offline dictionary text of the corresponding object of data manipulation Part.
In the present embodiment, when source database executes data manipulation, the definition information of object may change.For Guarantee the accuracy of the definition information of object, the present embodiment also needs the data manipulation of parsing Database Systems, and more new data is grasped Make the offline dictionary file of corresponding object.
Specifically, data manipulation is DDL operation, wherein DDL operation includes creation object, modification object and deletion Object.
In an alternate embodiment of the invention, data manipulation is creation object.Refering to Fig. 3, step 103: parsing source database Data manipulation, the offline dictionary file for updating the corresponding object of data manipulation include the following steps:
Step 1131: whether the object for judging to determine the creation of source database is to need synchronous object.
In the present embodiment, the object of source database creation is to belong to the object newly increased, local existing offline word It does not include corresponding offline dictionary file in allusion quotation.First, it is determined that whether the object of source database creation is to need synchronous pair As, specifically, can according to it is aforementioned acquisition reference object define information SQL statement judge determination the creation object whether To need synchronous object.If needing synchronous object, 1132 are thened follow the steps, if not needing synchronous object, is then held Row step 1134.
Step 1132: if needing synchronous object, obtaining the definition information and creation Object Operations of the object of creation Corresponding log serial number.
In Database Systems, one affairs of every generation corresponding can distribute corresponding log serial number, to distinguish not Same operation log.In the present embodiment, definition information and the creation Object Operations for obtaining the object of creation first are corresponding Log serial number.
Step 1133: it is offline that the definition information of the object of creation and corresponding log serial number being integrated into first version Dictionary file, and the offline dictionary file of first version is stored in local disk.
The definition information of the object of creation and corresponding log serial number are integrated into the offline dictionary file of first version, And the offline dictionary file of first version is stored in local disk.The method for specifically establishing offline dictionary file please refers to step 102, details are not described herein.
Step 1134: if not needing synchronous object, then not updating offline dictionary.
If not needing synchronous object, the corresponding offline dictionary file of creation object is not established.
When the present embodiment mainly illustrates to create object, offline dictionary how is updated, with guarantee to be stored in offline dictionary The accuracy of object definition information and comprehensive.
In an alternate embodiment of the invention, data manipulation is modification object.Refering to Fig. 4, step 103: parsing source database Data manipulation, the offline dictionary file for updating the corresponding object of data manipulation include the following steps:
Step 1231: obtaining the definition information and the corresponding log serial number of modification Object Operations of modified object.
In the present embodiment, when modifying to object, the definition information of object can be modified.For example, object is table, In practical application scene, after modifying to table name or column name, the definition information of object also changes accordingly.At this point, obtaining Take the definition information and modification Object Operations corresponding log sequence in the journal file of Database Systems of modified object Row number.
Step 1232: by the definition information of modified object and corresponding log serial number be integrated into next version from Line dictionary file.
In the present embodiment, when updating offline dictionary file corresponding by modification object, a version is not covered Offline dictionary file, but the offline dictionary file of more new version is added to the end of the corresponding offline dictionary file of the object Tail.In order to distinguish the offline dictionary file of the same object different editions, the updated offline dictionary of configuration log Serial No. The version number of file.In log analysis, corresponding definition information can be obtained according to the version number of offline dictionary file.Hereafter It how is discussed in greater detail according to the version number of offline dictionary file, obtains and corresponding define information.
In an alternate embodiment of the invention, data manipulation is to delete object.Refering to Fig. 5, step 103: parsing source database Data manipulation, the offline dictionary file for updating the corresponding object of data manipulation include the following steps:
Step 1331: obtaining and delete the corresponding log serial number of Object Operations;
In the present embodiment, when some object be deleted after, obtain be deleted the corresponding offline dictionary file of object and Delete Object Operations corresponding log serial number in the journal file of Database Systems.In an alternate embodiment of the invention, using spy The different offline dictionary file of character mark has been deleted, in order to determine the object to delete according to spcial character in subsequent operation, and will The corresponding offline dictionary file of deleted object is removed from local disk, to save disk space.
Step 1332: judging to determine and delete whether the corresponding log serial number of Object Operations is less than origin log sequence number.
Specifically, source database is in the data synchronous point for receiving purpose client database, according to data synchronous point It determines origin log sequence number, and determines the data for needing to be synchronized according to log serial number.Specifically, origin log sequence Number can be source database and receive the synchronization request for the first time of purpose client database is, corresponding in source database log file Maximum log serial number, and set origin log sequence number for the maximum log serial number.
In the present embodiment, source database, which judges to determine, deletes whether the corresponding log serial number of Object Operations has been less than Beginning log serial number illustrates that the object has been deleted before data synchronize if being less than, i.e., deleted object does not need to synchronize To purpose client database, 1333 are thened follow the steps.If being not less than, it is possible to deposit at the time point that synchronous point to the object is deleted Synchronous data are being needed, then will retain offline dictionary file, again by the deleted object pair after completing data and synchronizing The offline dictionary file answered is deleted from local disk.
Step 1333: if being less than, the corresponding offline dictionary file of deleted object being deleted from local disk.
It is aforementioned that the process that offline dictionary is updated according to different data manipulation (DDL operation) is described in detail, with guarantee from The accuracy of offline dictionary file and comprehensive in line dictionary, so simultaneously provide for log analysis or data it is accurate and comprehensive Object definition information.
Step 11: obtaining the operation log in source database log file.
In specific application scenarios, firstly, obtaining the data synchronous point of purpose client database, and according to data synchronous point Determine that origin log sequence number, origin log sequence number can be source database and receive synchronizing for the first time for purpose client database When request, corresponding maximum log serial number in source database log file, and the maximum log serial number is arranged For origin log sequence number.
Then, the operation day that log serial number in source database log file is not less than origin log sequence number is obtained Will is carrying out log analysis to the operation log for meeting aforementioned condition.
Step 12: the log serial number according to operation log obtains corresponding object definition information from offline dictionary, and Operation log is parsed according to the object definition information got.
In the present embodiment, due in journal file there are a large amount of operation log, the corresponding operation of each operation log Object may be different, and therefore, source database parses operation log to obtain target operation object.In optional embodiment In, available operation object name, then corresponding offline dictionary file is obtained from offline dictionary according to operation object name, to obtain Take correct object definition information.
Specifically, by Such analysis it is found that each object may correspond to the offline dictionary file of multiple versions, in order to Determine the object to be operated corresponding object definition of punctum in operation.In the present embodiment, it obtains comprising target operation object The corresponding log serial number of the offline dictionary file of all versions of information is defined, and compares log serial number and the institute of operation log There is the corresponding log serial number of the offline dictionary file of version, determines that the offline dictionary of respective version is literary based on nearest principle is greater than Part, to obtain the definition information of target operation object.Wherein, based on be greater than nearest principle refer to, the corresponding log of operation log Sequence number is possible to the corresponding log serial number of offline dictionary file greater than multiple versions, then choose be greater than and with one of them The smallest offline dictionary file of difference of the log serial number of the corresponding offline dictionary file of version is as the offline word of target version Allusion quotation file.Then, correct object definition information is obtained from the offline dictionary file of target version.
In order to which the aforementioned acquisition of clearer explanation correctly defines the process of information, first carried out with a specific application scenarios It illustrates.
It is now assumed that the corresponding offline dictionary file of target operation object includes the offline dictionary file of two versions, the first edition This offline corresponding log serial number of dictionary file is l0, after data base synchronization service starting, definition is performed to the object Modification operation, then the definition information after meeting adding modification operation execution after the offline dictionary file of first version is corresponding offline The offline dictionary file of dictionary file, the i.e. second edition, the corresponding log serial number of the offline dictionary file of the second edition are l1
In database synchronization process, the synchronous sequence of mark data is come with log serial number, source database carries out data When synchronous, origin log Serial No. l0, i.e., from l0I.e. from beginning synchronisation source client database data.In source database journal text Assume that, there are three operation log, which is ll in part-1, second operation log pair The log serial number answered is ll1, the corresponding log serial number of third operation log is ll2, and ll-1< l0< ll1< l1< ll2
When starting synchronisation source client database data, the corresponding log serial number of more each operation log and origin log Sequence number, due to ll-1The operation log of mark is in database synchronization (corresponding l0) before, then do not analyze ll-1Respective operations;The same day Will analyzes ll1When, due to l0< ll1< l1, that is, second operation log occurs in the offline dictionary file of first version Corresponding moment point moment point corresponding with the offline dictionary file of the second edition is then based on being chosen at offline word greater than nearest principle L in allusion quotation file0The corresponding definition of corresponding operation object is analyzed;When log analysis analyzes ll2When, due to l1< ll2, That is, third operation log occurs after the corresponding moment point of the offline dictionary file of the second edition, then it is former recently based on being greater than Then it is chosen at l in offline dictionary file1The corresponding definition of object is analyzed.It can be seen that can be according to current operation log Log serial number is correct to obtain according to the offline dictionary file for being greater than nearest principle selection right version in offline dictionary Define information.
Step 13: the data that log parses will be carried out and be sent to purpose client database.
In the present embodiment, the data that log parses will be carried out and is sent to purpose client database, naturally it is also possible to will Operation log and the corresponding object definition information of each operation log are sent to purpose client database, and purpose client database will be grasped It is synchronous that SQL statement progress data are reduced into as log.It can specifically be designed according to actual conditions, here, being not specifically limited.
In practical application scene, in order to reduce data volume of the source database to destination database synchronization, improve same The efficiency of step.In an alternate embodiment of the invention, before synchronizing to data, judge that determination needs the object being synchronized in synchronization Whether point is deleted before, if the object is deleted, when data are synchronous, does not execute corresponding to the object Multiple DML operation logs carry out log analysis, meanwhile, the corresponding data of deleted object are without being synchronized to destination number According to library.It can reduce some data volumes in this way, improve synchronous efficiency.
Specifically, if the corresponding log serial number of offline dictionary file of the last one version of the object is less than foundation Synchronous point and the origin log sequence number of determination, it is determined that the object for needing to be synchronized is deleted before synchronous point, then The corresponding data of the object are without being synchronized to purpose client database.
Be different from the prior art, the present invention carried out by the way of log analysis data it is synchronous when, initially set up and include The offline dictionary of object definition information, the accessible offline dictionary when carrying out log analysis are established and are only wrapped in the prior art The complementary parsing of operation log containing local key message, not only ensure that the correct of the definition information of object in operation log Property, while the case where also avoid real time access database and influence the performance of database.
Embodiment 2:
Referring to Fig. 6, Fig. 6 is a kind of structural schematic diagram of device synchronous for data provided in an embodiment of the present invention. The device synchronous for data of the present embodiment includes one or more processors 61 and memory 62.Wherein, with one in Fig. 6 For processor 61.
Processor 61 can be connected with memory 62 by bus or other modes, to be connected as by bus in Fig. 6 Example.
Memory 62 can be used for storing non-as a kind of non-volatile computer readable storage medium storing program for executing synchronous based on data Volatibility software program, non-volatile computer executable program and module, as the synchronous method of the data in embodiment 1 with And corresponding program instruction.Processor 61 by operation be stored in memory 62 non-volatile software program, instruction and Module, the data thereby executing the various function application and data processing of the synchronous method of data, i.e. realization embodiment 1 are synchronous Method function.
Wherein, memory 62 may include high-speed random access memory, can also include nonvolatile memory, such as At least one disk memory, flush memory device or other non-volatile solid state memory parts.In some embodiments, it stores Optional device 62 includes the memory remotely located relative to processor 61, these remote memories can pass through network connection to place Manage device 61.The example of above-mentioned network includes but is not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
The method synchronous about data please refers to FIG. 1 to FIG. 5 and relevant verbal description here, repeating no more.
It is worth noting that in information exchange, implementation procedure between module, unit in above-mentioned apparatus and system etc. Hold, due to being based on same design with processing method embodiment of the invention, particular content be can be found in embodiment of the present invention method Narration, details are not described herein again.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of embodiment is can to lead to Program is crossed to instruct relevant hardware and complete, which can be stored in a computer readable storage medium, storage medium It may include: read-only memory (Read Only Memory, be abbreviated as ROM), random access memory (Random Access Memory is abbreviated as RAM), disk or CD etc..
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention Made any modifications, equivalent replacements, and improvements etc., should all be included in the protection scope of the present invention within mind and principle.

Claims (10)

1. a kind of synchronous method of data, which is characterized in that the synchronous method of the data includes:
Establish the offline dictionary comprising object definition information;
Obtain the operation log in source database log file;
Log serial number according to the operation log obtains corresponding object definition information from the offline dictionary, and according to The object definition information got parses the operation log;
It will carry out the data that log parses and be sent to purpose client database.
2. method of data synchronization according to claim 1, which is characterized in that it is described establish comprising object definition information from After line dictionary further include:
The data synchronous point of purpose client database is obtained, and determines origin log sequence number according to the data synchronous point;
It is described obtain source database log file in operation log include:
Obtain the operation log that log serial number in source database log file is not less than the origin log sequence number.
3. the synchronous method of data according to claim 2, which is characterized in that described to establish comprising object definition information Dictionary includes: offline
Obtain the object definition information and the corresponding current log of current time source database in source Database Systems dictionary Sequence number;
The object definition information and the current log sequence number are integrated into the offline dictionary file of first version, and will be described The offline dictionary file of first version is stored in local disk;
The data manipulation for parsing source database, updates the offline dictionary file of the corresponding object of the data manipulation.
4. the synchronous method of data according to claim 3, which is characterized in that the acquisition source Database Systems dictionary In object definition information and the corresponding current log sequence number of current time source database include:
Based on structured query language query source client database system dictionary;
It obtains corresponding with the definition information for the object that the structured query language matches and current time source database Current log sequence number;Wherein, defining information includes the one or more in schema name, object name and district location.
5. the synchronous method of data according to claim 3, which is characterized in that the data manipulation is creation object;
The data manipulation of the parsing source database, updates the offline dictionary file packet of the corresponding object of the data manipulation It includes:
Whether the object for judging to determine the source database creation is to need synchronous object;
If needing synchronous object, the definition information and the corresponding logged sequence of creation Object Operations of the object of creation are obtained Number;
The definition information of the object of creation and corresponding log serial number are integrated into the offline dictionary file of first version, and will The offline dictionary file of first version is stored in local disk.
6. according to the synchronous method of the described in any item data of claim 3~5, which is characterized in that the data manipulation is to repair Change object;
The data manipulation of the parsing source database, updates the offline dictionary file packet of the corresponding object of the data manipulation It includes:
Obtain the definition information and the corresponding log serial number of modification Object Operations of modified object;
The definition information of modified object and corresponding log serial number are integrated into the offline dictionary file of next version.
7. according to the synchronous method of the described in any item data of claim 3~5, which is characterized in that the data manipulation is to delete Except object;
The data manipulation of the parsing source database, updates the offline dictionary file packet of the corresponding object of the data manipulation It includes:
It obtains and deletes the corresponding log serial number of Object Operations;
Judge to determine whether the corresponding log serial number of the deletion Object Operations is less than the origin log sequence number;
If being less than, the corresponding offline dictionary file of the deleted object is deleted from local disk.
8. according to the synchronous method of the described in any item data of claim 3~5, which is characterized in that described according to the operation The log serial number of log obtains corresponding object definition information from the offline dictionary, and according to the object definition got Information carries out parsing to the operation log
The operation log is parsed to obtain target operation object;
Obtain the corresponding log serial number of the offline dictionary file of all versions of the definition information comprising the target operation object;
Compare the log serial number of operation log log serial number corresponding with all offline dictionary files of version;
Based on the offline dictionary file for determining respective version greater than nearest principle, to obtain the definition letter of the target operation object Breath.
9. the generation method of offline dictionary according to claim 1, which is characterized in that the object is table, index and view One or more of figure.
10. a kind of device synchronous for data, which is characterized in that including at least one processor;And with described at least one The memory of a processor communication connection;Wherein, the memory is stored with the finger that can be executed by least one described processor It enables, described instruction is arranged to carry out the synchronous method of data as described in any one of claims 1 to 9 by program.
CN201810860116.0A 2018-08-01 2018-08-01 Data synchronization method and device for data synchronization Active CN109189852B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810860116.0A CN109189852B (en) 2018-08-01 2018-08-01 Data synchronization method and device for data synchronization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810860116.0A CN109189852B (en) 2018-08-01 2018-08-01 Data synchronization method and device for data synchronization

Publications (2)

Publication Number Publication Date
CN109189852A true CN109189852A (en) 2019-01-11
CN109189852B CN109189852B (en) 2021-05-28

Family

ID=64937608

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810860116.0A Active CN109189852B (en) 2018-08-01 2018-08-01 Data synchronization method and device for data synchronization

Country Status (1)

Country Link
CN (1) CN109189852B (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110119406A (en) * 2019-04-09 2019-08-13 口碑(上海)信息技术有限公司 The checking method and device of real-time task record
CN110347746A (en) * 2019-06-13 2019-10-18 武汉达梦数据库有限公司 A kind of heterogeneous database synchrodata consistency desired result method and device
CN110727548A (en) * 2019-09-29 2020-01-24 上海英方软件股份有限公司 Continuous data protection method and device based on database DML synchronization
CN111159020A (en) * 2019-12-20 2020-05-15 武汉达梦数据库有限公司 Method and device applied to synchronous software test
CN111221909A (en) * 2019-12-31 2020-06-02 武汉达梦数据库有限公司 Database modification column synchronization method and device based on log analysis
CN111221907A (en) * 2019-12-31 2020-06-02 武汉达梦数据库有限公司 Database added column synchronization method and device based on log analysis
CN111241094A (en) * 2019-12-31 2020-06-05 武汉达梦数据库有限公司 Database deleted column synchronization method and device based on log analysis
CN111930828A (en) * 2020-05-29 2020-11-13 武汉达梦数据库有限公司 Data synchronization method and data synchronization system based on log analysis
CN112307118A (en) * 2020-09-30 2021-02-02 武汉达梦数据库有限公司 Method for guaranteeing data consistency based on log analysis synchronization and synchronization system
CN112612859A (en) * 2020-12-31 2021-04-06 上海英方软件股份有限公司 DDL analysis method and device based on log analysis
CN113761052A (en) * 2020-11-27 2021-12-07 北京沃东天骏信息技术有限公司 Database synchronization method and device
CN116881371A (en) * 2023-09-07 2023-10-13 北京逐风科技有限公司 Data synchronization method, device, equipment and storage medium

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101930443A (en) * 2009-06-22 2010-12-29 金蝶软件(中国)有限公司 Data service system and data service realization method
EP2302538A1 (en) * 2009-09-16 2011-03-30 Business Objects Software Ltd. Method and system for capturing change of data
CN103617176A (en) * 2013-11-04 2014-03-05 广东电子工业研究院有限公司 Method for achieving automatic synchronization of multi-source heterogeneous data resources
CN104881418A (en) * 2014-02-28 2015-09-02 阿里巴巴集团控股有限公司 Method and device for quickly reclaiming rollback space in MySQL
CN105005618A (en) * 2015-07-21 2015-10-28 杭州合众数据技术有限公司 Data synchronization method and system among heterogeneous databases
CN105243067A (en) * 2014-07-07 2016-01-13 北京明略软件系统有限公司 Method and apparatus for realizing real-time increment synchronization of data
CN105320680A (en) * 2014-07-15 2016-02-10 中国移动通信集团公司 Data synchronization method and device
CN107943979A (en) * 2017-11-29 2018-04-20 山东鲁能软件技术有限公司 The quasi real time synchronous method and device of data between a kind of database
US20180150540A1 (en) * 2016-11-29 2018-05-31 Sap Se Synchronization Mechanism for Serialized Data Log Replay in Database Systems

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101930443A (en) * 2009-06-22 2010-12-29 金蝶软件(中国)有限公司 Data service system and data service realization method
EP2302538A1 (en) * 2009-09-16 2011-03-30 Business Objects Software Ltd. Method and system for capturing change of data
CN103617176A (en) * 2013-11-04 2014-03-05 广东电子工业研究院有限公司 Method for achieving automatic synchronization of multi-source heterogeneous data resources
CN104881418A (en) * 2014-02-28 2015-09-02 阿里巴巴集团控股有限公司 Method and device for quickly reclaiming rollback space in MySQL
CN105243067A (en) * 2014-07-07 2016-01-13 北京明略软件系统有限公司 Method and apparatus for realizing real-time increment synchronization of data
CN105320680A (en) * 2014-07-15 2016-02-10 中国移动通信集团公司 Data synchronization method and device
CN105005618A (en) * 2015-07-21 2015-10-28 杭州合众数据技术有限公司 Data synchronization method and system among heterogeneous databases
US20180150540A1 (en) * 2016-11-29 2018-05-31 Sap Se Synchronization Mechanism for Serialized Data Log Replay in Database Systems
CN107943979A (en) * 2017-11-29 2018-04-20 山东鲁能软件技术有限公司 The quasi real time synchronous method and device of data between a kind of database

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110119406A (en) * 2019-04-09 2019-08-13 口碑(上海)信息技术有限公司 The checking method and device of real-time task record
CN110347746A (en) * 2019-06-13 2019-10-18 武汉达梦数据库有限公司 A kind of heterogeneous database synchrodata consistency desired result method and device
CN110727548A (en) * 2019-09-29 2020-01-24 上海英方软件股份有限公司 Continuous data protection method and device based on database DML synchronization
CN111159020A (en) * 2019-12-20 2020-05-15 武汉达梦数据库有限公司 Method and device applied to synchronous software test
CN111241094B (en) * 2019-12-31 2021-06-08 武汉达梦数据库有限公司 Database deleted column synchronization method and device based on log analysis
CN111221909A (en) * 2019-12-31 2020-06-02 武汉达梦数据库有限公司 Database modification column synchronization method and device based on log analysis
CN111221907A (en) * 2019-12-31 2020-06-02 武汉达梦数据库有限公司 Database added column synchronization method and device based on log analysis
CN111241094A (en) * 2019-12-31 2020-06-05 武汉达梦数据库有限公司 Database deleted column synchronization method and device based on log analysis
CN111221909B (en) * 2019-12-31 2021-05-28 武汉达梦数据库股份有限公司 Database modification column synchronization method and device based on log analysis
CN111930828A (en) * 2020-05-29 2020-11-13 武汉达梦数据库有限公司 Data synchronization method and data synchronization system based on log analysis
CN111930828B (en) * 2020-05-29 2024-01-19 武汉达梦数据库股份有限公司 Data synchronization method and data synchronization system based on log analysis
CN112307118A (en) * 2020-09-30 2021-02-02 武汉达梦数据库有限公司 Method for guaranteeing data consistency based on log analysis synchronization and synchronization system
CN112307118B (en) * 2020-09-30 2024-03-22 武汉达梦数据库股份有限公司 Method for guaranteeing data consistency based on log analysis synchronization and synchronization system
CN113761052A (en) * 2020-11-27 2021-12-07 北京沃东天骏信息技术有限公司 Database synchronization method and device
CN112612859A (en) * 2020-12-31 2021-04-06 上海英方软件股份有限公司 DDL analysis method and device based on log analysis
CN116881371A (en) * 2023-09-07 2023-10-13 北京逐风科技有限公司 Data synchronization method, device, equipment and storage medium
CN116881371B (en) * 2023-09-07 2023-11-14 北京逐风科技有限公司 Data synchronization method, device, equipment and storage medium

Also Published As

Publication number Publication date
CN109189852B (en) 2021-05-28

Similar Documents

Publication Publication Date Title
CN109189852A (en) A kind of method that data are synchronous and the device synchronous for data
CN108932282B (en) Database migration method and device and storage medium
CN104657382B (en) Method and apparatus for MySQL principal and subordinate&#39;s server data consistency detections
US5613113A (en) Consistent recreation of events from activity logs
US10031935B1 (en) Customer-requested partitioning of journal-based storage systems
CN102402596B (en) A kind of reading/writing method of master slave separation database and system
CN106933703B (en) Database data backup method and device and electronic equipment
US9660962B2 (en) Network-attached storage gateway validation
US9418094B2 (en) Method and apparatus for performing multi-stage table updates
CN105574187B (en) A kind of Heterogeneous Database Replication transaction consistency support method and system
US10324905B1 (en) Proactive state change acceptability verification in journal-based storage systems
CN105205053A (en) Method and system for analyzing database incremental logs
WO2022002103A1 (en) Method for playing back log on data node, data node, and system
CN105808653A (en) User label system-based data processing method and device
US20020091716A1 (en) Replication system and program
US10235407B1 (en) Distributed storage system journal forking
US20150081745A1 (en) Database insert with deferred materialization
CN110489092B (en) Method for solving read data delay problem under database read-write separation architecture
CN114741453A (en) Method, system and computer readable storage medium for data synchronization
JP2011013792A (en) Device, method and program for control of database in program model inspection
CN107885824A (en) A kind of data read-write method and system
WO2012056734A1 (en) Database management method
CN110647421B (en) Database processing method, device and system and electronic equipment
US11789971B1 (en) Adding replicas to a multi-leader replica group for a data set
JP2007264814A (en) Program for creating replication data, program for reflecting replication data, and database device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB03 Change of inventor or designer information
CB03 Change of inventor or designer information

Inventor after: Fu Quan

Inventor after: Sun Feng

Inventor after: Chen Yan

Inventor before: Fu Quan

Inventor before: Sun Feng

Inventor before: Chen Yan

Inventor before: Yang Chun

GR01 Patent grant
GR01 Patent grant
CP01 Change in the name or title of a patent holder
CP01 Change in the name or title of a patent holder

Address after: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province

Patentee after: Wuhan dream database Co.,Ltd.

Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province

Patentee before: WUHAN DAMENG DATABASE Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20220908

Address after: 430073 16-19 / F, building C3, future science and technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province

Patentee after: Wuhan dream database Co.,Ltd.

Patentee after: HUAZHONG University OF SCIENCE AND TECHNOLOGY

Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province

Patentee before: Wuhan dream database Co.,Ltd.

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20230728

Address after: 16-19/F, Building C3, Future Science and Technology Building, No. 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province, 430206

Patentee after: Wuhan dream database Co.,Ltd.

Address before: 430073 16-19 / F, building C3, future science and technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan City, Hubei Province

Patentee before: Wuhan dream database Co.,Ltd.

Patentee before: HUAZHONG University OF SCIENCE AND TECHNOLOGY