CN108363791A - A kind of method of data synchronization and device of database - Google Patents
A kind of method of data synchronization and device of database Download PDFInfo
- Publication number
- CN108363791A CN108363791A CN201810150626.9A CN201810150626A CN108363791A CN 108363791 A CN108363791 A CN 108363791A CN 201810150626 A CN201810150626 A CN 201810150626A CN 108363791 A CN108363791 A CN 108363791A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- string
- log information
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Abstract
The invention discloses a kind of method of data synchronization of database and devices, by reading the log information not synchronous in first database with parsing, the data information to be synchronized can be obtained from first database, to which the data information can be synchronized to the second database.Therefore, if using operation system database as first database, operation system database may not necessarily search the data information to be synchronized to carry out data synchronization to other databases, the database for avoiding operation system in this way searches task in face of the mass data generated when data synchronization, so that frequently even real-time data synchronous task does not interfere with processing capacity of the database to data processing task caused by operation system operational process of operation system, to realize the real-time data synchronization of database in the case where the normal operation of operation system is unaffected.
Description
Technical field
The present invention relates to database technical fields, more particularly to the method for data synchronization and device of a kind of database.
Background technology
With the development of database technology, more and more operation systems are stored and are managed to data using database
Reason.During operation system is run, generated data are recorded in the database of operation system.And for the ease of industry
Business side is analyzed and is handled to data, and the data of operation system database need to be synchronized to other databases of business side, with
Form the backup of the data in operation system.In the prior art, the database synchronization data from operation system is needed each time
When to another database, the database of operation system is required for searching the data to be synchronized.But some are wanted
It asks data to synchronize frequent scene, especially requires the scene of real-time data synchronization, excessively frequent data synchronization that can lead to industry
The database of business system faces a large amount of data search task, and the database to influence operation system runs operation system
The processing capacity of data processing task caused by process so that the normal operation of operation system is affected.
Invention content
The technical problem to be solved by the invention is to provide a kind of method of data synchronization of database and device, to avoid
The mass data that the database of operation system generates when being synchronized in face of data searches task so that frequent even real-time data
Synchronous task does not interfere with place of the database to data processing task caused by operation system operational process of operation system
Reason ability, to realize the real-time data synchronization of database in the case where the normal operation of operation system is unaffected.
In a first aspect, an embodiment of the present invention provides a kind of method of data synchronization of database, including：
In response to the triggering command that data synchronize, the log information not synchronized is read from first database；
The log information is parsed to obtain target string；
According to the format of the log information, the target string is converted into target data information；
The target data information is synchronized to the second database.
Optionally, described that the log information is parsed to obtain target string, including：
If the corresponding data manipulation of the log information is insertion operation, fixed-length string is searched from the log information
And/or varying string；
The fixed-length string and/or the varying string are determined as the target string.
Optionally, described that the log information is parsed to obtain target string, including：
It is operated if the corresponding data manipulation of the log information is update, the update is searched from the log information and is grasped
Make corresponding more new character strings；
Obtain the corresponding init string of primary data information before the update operation；
The init string is updated with the more new character strings, obtains the target string.
Optionally, the corresponding init string of primary data information obtained before the update operation, including：
Fixed length major key and/or elongated major key are searched from the log information；
According to the fixed length major key and/or elongated major key, the primary data information is searched in second database；
The primary data information is converted into the init string.
Optionally, the first database is SQL Server databases.
Second aspect, an embodiment of the present invention provides a kind of data synchronization units of database, including：
Reading unit, the triggering command for being synchronized in response to data, reads the day not synchronized from first database
Will information；
Resolution unit, for being parsed to obtain target string to the log information；
Converting unit, for according to the log information format, the target string is converted into number of targets it is believed that
Breath；
Synchronization unit, for the target data information to be synchronized to the second database.
Optionally, the resolution unit includes：
First searches subelement, if being insertion operation for the corresponding data manipulation of the log information, from the daily record
Fixed-length string and/or varying string are searched in information；
Determination unit, for the fixed-length string and/or the varying string to be determined as the target string.
Optionally, the resolution unit includes：
Second searches subelement, if being that update operates for the corresponding data manipulation of the log information, from the daily record
The update is searched in information operates corresponding more new character strings；
Subelement is obtained, for obtaining the corresponding init string of primary data information before the update operates；
Update subelement obtains the target for being updated to the init string with the more new character strings
Character string.
Optionally, the acquisition subelement includes：
Third searches subelement, for searching fixed length major key and/or elongated major key from the log information；
4th searches subelement, for according to the fixed length major key and/or elongated major key, being looked into second database
Look for the primary data information；
Conversion subunit, for the primary data information to be converted into the init string.
The third aspect, an embodiment of the present invention provides a kind of data synchronizer of database, the equipment includes processing
Device and memory：
Said program code is transferred to the processor by the memory for storing program code；
The data that the processor is used to be provided according to the above-mentioned first aspect of instruction execution in said program code
The method of data synchronization in library.
Fourth aspect, an embodiment of the present invention provides a kind of storage medium, the storage medium is used to store program code,
Said program code is used to execute the method for data synchronization for the database that above-mentioned first aspect provides.
Compared with prior art, the embodiment of the present invention has the following advantages：
In embodiments of the present invention, by reading the log information not synchronous in first database with parsing, Ke Yicong
The data information to be synchronized is obtained in first database, to which the data information can be synchronized to the second database.Therefore,
If using operation system database as first database, operation system database may not necessarily be in order to other databases into line number
The data information to be synchronized is searched according to synchronous, avoids the database of operation system in this way in face of being generated when data synchronization
Mass data search task so that frequent even real-time data synchronous task does not interfere with the database of operation system
To the processing capacity of data processing task caused by operation system operational process, to the normal operation of operation system not by
The real-time data synchronization of database is realized in the case of influence.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments described in invention, for those of ordinary skill in the art, without creative efforts,
Other drawings may also be obtained based on these drawings.
Fig. 1 is the network system schematic diagram of an exemplary application scene in the embodiment of the present invention；
Fig. 2 is a kind of flow chart of the method for data synchronization of database provided in an embodiment of the present invention；
Fig. 3 is the flow chart that a kind of parsing update provided in an embodiment of the present invention operates corresponding log information；
Fig. 4 is a kind of method of data synchronization scene embodiment flow chart of database provided by the invention；
Fig. 5 is a kind of structure chart of the data synchronization unit of database provided in an embodiment of the present invention；
Fig. 6 is a kind of hardware architecture diagram of the data synchronizer of database provided in an embodiment of the present invention.
Specific implementation mode
In order to enable those skilled in the art to better understand the solution of the present invention, below in conjunction in the embodiment of the present invention
Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only this
Invention a part of the embodiment, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art exist
The every other embodiment obtained under the premise of creative work is not made, shall fall within the protection scope of the present invention.
The more and more data that will produce during operation system is run, these data can be by the operation systems
Database is to the relevant record of these data and processing.Currently, data are analyzed and handled for the ease of business side, need
The data that will be recorded in operation system database are synchronized in other databases of the business side.And every time from operation system
Database synchronization data to another database of business side, be required for the database of operation system to expend certain performance resource
To search the data to be synchronized.And in order to not influence the normal operation of operation system, above-mentioned data simultaneously operating can according to compared with
The long period executes, and carries out a data within such as one day and synchronizes, also, the execution time of data simultaneously operating can generally also select industry
The period of business systematic comparison free time, such as at night.
But with the arrival in big data epoch, be required in many scenes data can continually, even in real time by
It is synchronous, will pass through the analysis to real-time synchronization data and processing, realize momentarily, accurately to business in the operation system
Analysis and prediction.And due to another database every time from the database synchronization data of operation system to business side, it is required for
The database of operation system expends certain performance resource to search the data to be synchronized, so, realizing that real time data is same
The database of operation system is just needed to expend resource to carry out search operation when step.And the database of operation system is frequently executed
The performance of search operation, operation system database will reduce, then inevitably result in the database of the operation system to business system
The processing capacity of data caused by system operational process is affected, and then influences whether the normal operation of the operation system.
Based on this, frequently executes search operation in order to solve the operation system database in database real-time synchronization and bring
Database where the impacted problem of operation system normal operation, an embodiment of the present invention provides a kind of data of database
Synchronous method and apparatus, not only may be implemented the real-time data synchronization of database, but also may insure not influence operation system just
Often operation.
It, can be from the in the case where being responsive to the triggering command of data synchronization specifically, in embodiments of the present invention
The log information not synchronized is read in one database, it is then possible to the log information is parsed to obtain target string,
According still further to the format of log information, which can be converted to target data information, so as to by target data
Synchronizing information is to the second database.
It follows that in embodiments of the present invention, it, can when operation system database carries out data synchronization to other databases
With by read with parse the log information in operation system database obtain will be synchronous data, without business system
System database goes to search the data information to be synchronized, and avoids operation system database in this way and is executed when data synchronize greatly
Data look-up operations are measured, place of the database to data produced by operation system operational process of operation system would not be also influenced
Reason ability, to realize the real-time data synchronization of database in the case where the normal operation of operation system is unaffected.
For example, the embodiment of the present invention can be applied to scene as shown in Figure 1.In this scenario, including first counts
According to library 101, the second database 102 and sync server 103.Wherein, first database 101 and the second database 102 are used
In being stored, analyzed and handled to data.Sync server 103 is counted with first database 101 and second respectively by network
Data interaction is carried out according to library 102, realizes the data in first database 101, is synchronized in the second database 102.
Specifically data synchronization process is：First, the triggering command that sync server 103 is synchronized in response to data, from
The log information not synchronized is read in one database 101；Then, sync server 103 to the log information parse
To target string；Then, target string is converted to target data by sync server 103 according to the format of log information
Information；Finally, which is synchronized to the second database 102 by sync server 103.It is understood that first
Database 101 can be the database that storage service data are used in operation system, and the second database 102 corresponds to for operation system
Business side database.Wherein, operation system refers to the system for providing certain business function to the user, such as " railway
12306 ", as provide a user the operation system of purchase train ticket service.It should be noted that above application scene be only for
It is easy to understand the present invention and shows, embodiments of the present invention are unrestricted in this regard.On the contrary, the implementation of the present invention
Mode can be applied to applicable any scene.
It should be noted that the sync server 103 can independently arrange in a network, the first number can also be integrated in
According in the system where library 101 or the second database 102.
Below in conjunction with the accompanying drawings, the various non-limiting embodiments that the present invention will be described in detail.
It is a kind of flow chart of the method for data synchronization of database provided in an embodiment of the present invention referring to Fig. 2.In this implementation
In example, the method can specifically include following steps：
Step 201, the triggering command synchronized in response to data reads the daily record letter not synchronized from first database
Breath.
Wherein, first database, such as can be the database in operation system, it is used for record traffic system operation
The data of middle generation, while also these data are carried out with certain processing.
It is understood that first database for example can be SQL Server databases.SQL Server databases are
A kind of relevant database can provide safe and reliable store function for relational data and structural data.SQL
The advantages of Server databases is, easy to use, scalability is good and related software degree of integration height etc..
It should be noted that when user carries out a data manipulation in operation system, which will correspond to production
Raw at least one log information corresponding with the data manipulation, the data generated under the log information recording data manipulation are believed
Breath.In SQL Server databases, data information can be converted into the character string of 16 systems, be stored in log information.
There are Current LSN fields, field expression to work as SQL Server databases, in log information
Preceding log serial number, each log information are identified by a unique log serial number (LSN).The specific sortord of LSN
For：If LSN2 is more than LSN1, the corresponding change of the log information that LSN2 is identified is happened at the letter of the daily record represented by LSN1
After ceasing corresponding change.
In the case where first database is SQL Server databases, the reading of log information can be in first database
It is realized by function fn_dblog.Specifically, when data synchronize, sync server is sent out to the first database of operation system
After sending function fn_dblog, first database to receive function fn_dblog, the function is run, you can read first data
The log information not synchronized in library.
It is understood that when data synchronize, the log information read from first database is typically not have before
The log information synchronized.It, can be by remembering in log information in the case where first database is SQL Server databases
The LSN of record determines whether log information synchronized, to reading the log information not synchronized.It, can be with when specific implementation
LSN after the corresponding LSN of log information that above data synchronize is corresponded to as this log information for needing data to synchronize
LSN value ranges, thus can read the daily record not synchronized in the first database according to the LSN value ranges
Information.
As an example it is assumed that the maximum LSN of log information is 150 in first database, after a data manipulation,
Generate the log information that LSN is 151,152 and 153.When needing to carry out data synchronization, sync server is by function fn_
After dblog is sent to first database, first database runs function fn_dblog, this obtained needs data to synchronize
The log information that log information, as LSN are 151,152 and 153.
Step 202, log information is parsed to obtain target string.
In log information, the character string that the data information generated by data manipulation is converted into is had recorded, which can
To be denoted as target string.
If log information corresponds to different data manipulations, log information is to the recording mode of target string generally also not phase
With for example, in the case where first database is SQL Server databases, log information contains following field：
Operation：The data manipulation that current LSN is done；
Row Log Contents 0：The raw information of insertion operation is had recorded, the data before update operation (only include
The part of modification)；
Row Log Contents 1：Have recorded the data (only being changed relative to old value) after update operation；
Row Log Contents 2：Have recorded major key information when update operation；
Row Log Contents 3：In Log Record fields, the information after the field is to update operation phase
The information of pass；
Row Log Contents 4：It has recorded update and operates relevant information；
Log Record：The control information for having recorded some length and location index when update operation, contains Row
Log Contents 0, Row Log Contents 1, Row Log Contents 2, Row Log Contents 3, Row
Information in 4 each fields of Log Contents.
Wherein, the data manipulation (i.e. the content of Operation fields) that current LSN is done, as shown in table 1 below.
The common explanation of 1 data manipulation of table
| The data manipulation (Operation) that current LSN is done | It explains |
| LOP_INSERT_ROWS | It is inserted into data |
| LOP_DELETE_ROWS | Delete data |
| LOP_MODIFY_ROW | Change certain a line record |
| LOP_MODIFY_COLUMS | Change the more complex record of certain a line |
Specifically, data manipulation may include that insertion operation and update operate, and it may include deleting and repairing to update operation
Change two kinds of action types.Wherein, above-mentioned LOP_MODIFY_ROW and LOP_MODIFY_COLUMNS can be indicated two different
Update operation, the rwo will substantially make an amendment data information.It is understood that in log information to target string
Recording mode difference in the case of, the mode that target string is usually parsed from log information may also be different.Below
For the log information of different data operation, illustrate how that log information is parsed to obtain target string respectively.
In some embodiments, the case where being insertion operation for the corresponding data manipulation of log information, step 202 can be with
It specifically includes：If the corresponding data manipulation of log information is insertion operation, fixed-length string and/or change are searched from log information
Long character string；Fixed-length string and/or varying string are determined as target string.
Wherein, insertion operation refers to having increased a data newly in first database.The data is in this insertion operation
Before, it is not present in first database.The insertion operation, the data information that the data of generation include, will be converted into word
Symbol string is recorded in a log information, which is the corresponding log information of the insertion operation.That is, in log information
The corresponding character string of data information of record is exactly the corresponding target string of the insertion operation.
When data manipulation is insertion operation, in generated log information, the data information of insertion is directly by with fixed length
The form of character string and/or elongated target character, is stored in log information.That is, the data information being inserted into directly corresponds to daily record
Partial character string in the character string of information, the partial character string are denoted as target string.
As an example it is assumed that performing an insertion operation, data A, including data information are inserted into first database
Abc illustrates before this insertion operation, does not include data A in first database；And the corresponding daily record letter of the insertion operation
In breath, data information abc is indicated with lookup fixed-length string and/or varying string, then it represents that the fixed length of data information abc
Character string and/or varying string are target string.
With a specific example, the analysis mode of the log information of insertion operation generation is introduced.
In this example, if first database is SQL Server databases, first database executes following sentence, with reality
The insertion operation of existing a data：
INSERTINTOuser1(id,sex,name,photo,describe,age,weight,yn,birthday,
birthday2,age2,college,own,own2,own3,character,address)VALUES(1,0,'zhangsan',
Null, null, 22,123.44,1, GETDATE (), GETDATE (), 3453, ' northeast big ', and null, 2342.234,
534.23, ' laugh a great ho-ho ', ' heartily ').
For log information caused by above-mentioned insertion operation, parsed using following sentence：
SELECT a.[Current LSN],a.Operation,a.Context,a.[Row Log Contents 0],
a.[Row Log Contents 1],a.[Row Log Contents 2],a.[Row Log Contents 3],a.[Row
Log Contents 4],a.[Log Record]FROM fn_dblog(null,null)
AWHEREa.allocunitnameLIKE'%user1%'ANDOperationIN (' LOP_INSERT_ROWS', ' LOP_
DELETE_ROWS','LOP_MODIFY_ROW','LOP_MODIFY_COLUMNS')ORDERBYa.[Current LSN]
DESC。
The content of 0 fields of Row Log Contents can be obtained after parsing, it is specific as follows：
Row Log Contents 0=
0x3000B000010000001816005C8FC2F528DC5E400160339D0037A300003C0237A37D0D0000000
00000608FE30E0000000001EF9203000000000001D62608000000000075547554200020002000
20002000200020002000200020002000200020002000200020002000200020002000200020002
00020002000200020002000B9FEB9FE2020202020202020202020202020202020202020202020
202020202020202020202020202020202020202020202011001810000400C700C700C700CF007
A68616E6773616E1C4E17532759665B。
2 are see the table below particularly with regard to the composition brief description of 0 fields of Row Log Contents.
The brief description that 2 Row Log Contents of table, 0 fields are constituted
It should be noted that is recorded in the log information is the character string of 16 systems, wherein every two character representations one
A byte, is divided into high-low-position, and low level is preceding, and a high position is rear.
It is as follows that the solution of Row Log Contents 0 specifically analyses flow：
(1), by the 3rd byte, can getting the address A of number of columns, (176) B000, the corresponding decimal system are.
(2), the columns N (1100, the corresponding decimal system 17) of table is obtained by the result in (1).
(3), K byte-identifier null value row are needed as a result, calculating by (2), K is intercepted since the A+2 of address
Then which byte obtains and is classified as sky.(each byte 8, can at most indicate 8 fields, and the table 17 fields in total need
Three bytes indicate, i.e., 181000, corresponding binary system is 1000000011000, totally 13, needs to mend 40 in front,
I.e. 00001000000011000, see from back to front, 0 is has value, and 1 is sky).
The a data being inserted into first database is incorporated in it is found that being finally inserted into the data in database, arranges photo,
Describe, own are empty (null), other row have occurrence.
(4), table structural information is obtained, is known which field is non-empty according to the result of (3), according to the word of field
Joint number intercepts corresponding length successively since the 5th byte, they are followed successively by the value of the fixed-length field in table, then parse
The hexadecimal log information.
That is, by the hexadecimal string of overstriking, is first intercepted into section according to length, be then parsed into tool by every section
The decimal value of body, the value that can obtain each row are how many.
(5), the corresponding character string of the data of variable-length field is stored in the end of log information.Since the addresses A+2+K, cut
Two bytes are taken, the number N of variable-length field is obtainedIt is elongated。
Specifically, it is 4 to be parsed from 0x0400, that is, there are four elongated row in total.
(6), two bytes are intercepted every time, until having intercepted 2NIt is elongatedUntil a byte, the byte of these interceptions stores successively
The end address of each variable-length field hexadecimal data.
For example, 0xC700 parses, to be parsed for 199,0xCF00 be 207.
(7), the end address for parsing the variable-length field got in (6), obtains the hexadecimal of corresponding variable-length field
Data simultaneously parse.
Such as：The hexadecimal string 7A68616E6773616E1C of italic is changed into zhangsan.
By above step, a data information being simply inserted into can be parsed.
In further embodiments, it is the case where update operates for the corresponding data manipulation of log information, such as Fig. 3 institutes
Show, step 202 can specifically include：
Step 301, it is operated if the corresponding data manipulation of log information is update, update operation pair is searched from log information
The more new character strings answered；
Step 302, the corresponding init string of primary data information before update operation is obtained；
Step 303, init string is updated with more new character strings, obtains target string.
Wherein, update operation refers in first database at least one of existing a data data information
It is modified or delete operation, the data information in the data is changed.In the case of one kind, data are grasped in update
Include data information abc before making, includes data information adc after update operation, that is, data information b is revised as data
Information d.In another case, data include data information abc before update operates, include that data are believed after update operation
Cease ac, that is, delete data information b.
In some log informations, when data manipulation is update operation, in generated log information, the pass that is recorded
Corresponding character string is operated in the update, can be used for illustrating the situation of change of data information in data, without recording data
Data information after operation, that is, in this case, can there was only more new character strings without target string in log information.Example
Such as, before update operation, it includes data information abc which, which operates corresponding data, after update operates, the data packet
Data information adc is included, then, in the log information, including more new character strings, the variation for recording " data a → data d ",
Rather than the data information adc in the current data.
It, can be by inquiring the data corresponding relevant dates all in first database after obtaining more new character strings
Will information, according to the time sequencing of log information, iteration show that the update operates corresponding target string, in this way, every time more
It newly operates corresponding data to synchronize, can all carry out a large amount of search operation to first database, influence the normal of first database
Data processing.
As a kind of concrete implementation mode, in order to not influence the normal work of first database, reduces data and synchronize pair
The occupancy of first database resource, moreover, in order to which the efficiency for improving acquisition target string can after obtaining more new character strings
By searching the corresponding primary data information of the data in the second database, to obtain init string, utilize initial word
Symbol string and more new character strings obtain target string.Specifically, step 302 can specifically include：It is searched from log information
Fixed length major key and/or elongated major key；According to fixed length major key and/or elongated major key, in the second database search initial number it is believed that
Breath；Primary data information is converted into init string.
Wherein, major key is the data directory of every data in database, in the database, can be with according to data directory
Uniquely find a data corresponding with the data directory.It should be noted that first database and the second database, for
The major key or data directory of same data are consistent.
It is understood that the character string of log information includes the fixed length major key of the data and/or elongated major key.When
For sync server after reading the corresponding log information of update operation in first database, which can be from this
Fixed length major key and/or elongated major key are found in the character string of log information.Then, in the second database, according to the fixed length
Major key and/or elongated major key find the fixed length major key and/or the corresponding a data of elongated major key, as initial number it is believed that
Breath.That is, the primary data information, that corresponding data includes before being updated operation with first database
Data information it is consistent.Finally, according to the rule that data information is converted to character string in log information, by the initial number it is believed that
Breath is converted to init string.
After obtaining more new character strings and init string, init string can be updated with more new character strings,
Obtain target string.
As an example it is assumed that in the log information read from first database for data A, obtain for indicating " number
According to the more new character strings 1 of b → data d " variations；Then, the corresponding primary data information of searching data A in the second database
Abc, and by format conversion obtains the corresponding init strings of primary data information abc 2；Finally, more new character strings 1 are utilized
Corresponding information " b → d " is updated 2 corresponding data information abc of init string, obtains target string 3, is used for
Recording data information adc.
With a specific example, the analysis mode of the log information of update operation generation is introduced.
In this example, if first database is SQL Server databases.Wherein, update operation is divided into two kinds of situations：
The update operation of the first situation is LOP_MODIFY_ROW；The update operation of the second situation is LOP_MODIFY_
COLUMNS。
For the first situation, the update action statement that first database executes is：Updateuser1set-name='
Lisi'whereid=1.Operating generated log information for the update is：
Operation=LOP_MODIFY_ROW；
Row Log Contents 0=0xC700C700C700CF007A68616E6773616E；
Row Log Contents 1=0xC300C300C300CB006C697369；
Row Log Contents2=0x1601000000010000；
Row Log Contents 3=0x0101000C0000E0F387350000010200040204000A0100864
70766；
Row Log Contents 4=0x；
Log Record=0x00003E004F000000EB10000001000200574900000000040 29400000
001000000FE0000004F000000DF100000440000010000A70000000001B7001000000004001000
0C0008001A00C700C700C700CF007A68616E6773616EC300C300C300CB006C697369160100000
00100000101000C0000E0F387350000010200040204000A0100864707661600。
In this case, we are after the hexadecimal string for getting whole data, it is only necessary to simply replace
Change corresponding hexadecimal string.
For the second situation, the update action statement that first database executes is：Updateuser1setsex=1,
Name='wang wu'whereid=1.Operating generated log information for the update is：
Operation=LOP_MODIFY_COLUMNS；
Row Log Contents 0=0x08000800B700B700；
Row Log Contents 1=0x01000C00；
Row Log Contents 2=0x1601000000010000；
Row Log Contents 3=0x0101000C0000E0F387350000010200040204000A0100864
70766；
Row Log Contents 4=0x1800864719661600C300C300C300CB006C697369C500C50
0C500CD0077616E677775AD00；
Log Record=0x00003E004F000000EC10000001000200584900000000060 29400000
001000000FE0000004F000000EB100000020000010000A7000000000100000000000008000800
040008001A00010001000C000E0008000800B700B70001000C0016010000000100000101000C0
000E0F387350000010200040204000A010086470766000A1800864719661600C300C300C300CB
006C697369C500C500C500CD0077616E677775AD00。
Wherein, due to containing the letter of Row Log Contents 0~Row Log Contents 4 in Log Record
Breath, so, some explanations are done just for Log Record now.About the composition of Log Record, in see the table below 3 briefly
It is bright.
The brief description that 3 Log Record of table are constituted
Wherein, the specific process of analysis of Log Record is as follows：
(1), by the 3rd, 4 two byte, the address A for storing newer data information can be obtained.
(2), what is stored at the byte of address A~A+10 is some flag informations, to the solution of the log information of update operation
It analyses useless, directly skips.
(3), cycle interception Row Log Contents 0, intercept two bytes, before being followed successively by the update at i-th every time
First address, updated first address at i-th；At the same time, since at A+11, Log Record are intercepted, every time interception two
Byte, the byte number being followed successively by before being updated at i-th, updated byte number at i-th.
It should be noted that only having recorded the length before update in Row Log Contents 1, updated length can only
It is obtained from Log Record.
(4), according to first database table structure, the related definition of major key is sequentially obtained, e.g., major key field name,
Type, length etc..Since at the 2nd byte of Row Log Contents 2, corresponding length is intercepted successively, it is corresponding to be
The value of each fixed length major key；After the completion of the value interception of fixed length major key, then 2 bytes are intercepted, this 2 byte storages are major keys
Total quantity；Then 1 byte is skipped, then intercepts 2 bytes, that the two bytes store is the quantity N of elongated major keyElongated major key；So
After recycle successively interception 2 bytes, respectively correspond to i-th of elongated major key end address, until having intercepted NElongated major keyIt is a；Then
Further according to each end address, the corresponding hexadecimal string of each elongated major key is intercepted successively.
(5), according to the major key obtained in (4), to corresponding data are inquired in the second database, as initial number it is believed that
Breath, and these primary data information are converted into hexadecimal string, it is denoted as init string.
(6), since the 1st byte in Row Log Contents 4, the byte before being updated at i-th is intercepted successively,
Byte (integral multiple divided by 4 round up that byte number is 4) before being updated at completion i-th, updated byte at i-th, completion
Updated byte at i-th (integral multiple divided by 4 that byte number is 4 round up).
(7), in the result obtained from (5) (6), corresponding byte is replaced.
By above step, one update can be parsed and operate corresponding target string.
In addition, for the specific implementation of step 302, can also be：It records and is carried out every time more in sync server
Log information after new operation, and when parsing new update operation, the data pair is directly obtained from the sync server
The log information answered recycles more new character strings to update init string from init string is wherein obtained, and obtains target character
String.
The parsing to log information can may be implemented by any of the above realization method in the present invention, to obtain
Which kind of realization method target string specifically uses, is not especially limited in the present embodiment.
Step 203, according to the format of log information, target string is converted into target data information.
In first database, using certain rule, data information is converted into character string, is recorded in log information
In.Similarly, according to identical rule, the character string in log information can be converted to data information.For example, for target word
Symbol string 3, according to the transition form of log information and data information, target data information adc is converted to by the target string 3.
Wherein, target data information may include：The Chinese phonetic alphabet, letter and number etc..
Step 204, the target data information is synchronized to the second database.
In the case of one kind, when data manipulation is insertion operation, step 204 can be specifically, by target data information pair
The corresponding position in the second database should be inserted into.
In another case, when data manipulation is update operation, step 204 can be specifically, with the number of targets of acquisition
It is believed that breath replaces the corresponding primary data information (that is, the corresponding data information of init string) in the second database.
In embodiments of the present invention, by reading the log information not synchronous in first database with parsing, Ke Yili
With format conversion by the target string in log information be converted to needed for synchronous target data information, to can should
Target data information is synchronized to the second database, and the data for realizing first database to the second database synchronize.As it can be seen that business system
First database in system may not necessarily be in order to carry out data synchronization to other databases, and search the data to be synchronized letter
Breath avoids the first database of operation system in face of executing mass data search operation when data synchronization so that frequency in this way
Numerous even real-time data simultaneously operating does not interfere with the first database of operation system to operation system operational process institute
The ability that the data of generation are handled, to realize database in the case where the normal operation of operation system is unaffected
Real-time data synchronization.
After the method for data synchronization for describing database provided in an embodiment of the present invention, below with a concrete scene
The form of embodiment introduces the method for data synchronization of database provided by the invention from the angle of Signalling exchange in conjunction with Fig. 4.
In the concrete scene, the data of the database, which synchronize, to be needed：Operation system 101, sync server 103 and industry
Business side 102.Wherein, between 102 sync server 103 of between operation system 101 and sync server 103 and business side
Data interaction can be carried out by internet.Moreover, operation system 101 includes first database；Business side 102 includes
Two databases.
Fig. 4 is a kind of flow chart of the method for data synchronization of database in the present embodiment, the method may include：
Step 401, sync server 103 sends function fn_dblog to the first database of operation system 101.
Step 402, the first database of operation system 101 runs function fn_dblog, obtains not same in first database
The log information walked.
Step 403, the log information not synchronized in the first database of acquisition is sent to synchronous clothes by operation system 101
Business device 103.
Step 404, sync server 103 parses the log information received, obtains target string.
Step 405, the target string of acquisition is converted, is obtained according to the format of log information by sync server 103
Obtain target data information.
Step 406, the target data information of acquisition is synchronized to the second database of business side 102 by sync server 103
In.
In the present embodiment, concrete implementation mode and the effect reached can be found in retouching for embodiment shown in Fig. 2
It states, which is not described herein again.
In addition, the embodiment of the present invention also additionally provides a kind of data synchronization unit of database, as shown in figure 5, being the dress
The structure chart set, the device include：
Reading unit 501 is read from first database and was not synchronized for the triggering command that is synchronized in response to data
Log information；
Resolution unit 502, for being parsed to obtain target string to the log information；
The target string is converted to target data by converting unit 503 for the format according to the log information
Information；
Synchronization unit 504, for the target data information to be synchronized to the second database.
Optionally, the resolution unit 502 includes：
First searches subelement, if being insertion operation for the corresponding data manipulation of the log information, from the daily record
Fixed-length string and/or varying string are searched in information；
Determination unit, for the fixed-length string and/or the varying string to be determined as the target string.
Optionally, the resolution unit 502 includes：
Second searches subelement, if being that update operates for the corresponding data manipulation of the log information, from the daily record
The update is searched in information operates corresponding more new character strings；
Subelement is obtained, for obtaining the corresponding init string of primary data information before the update operates；
Update subelement obtains the target for being updated to the init string with the more new character strings
Character string.
Optionally, the acquisition subelement includes：
Third searches subelement, for searching fixed length major key and/or elongated major key from the log information；
4th searches subelement, for according to the fixed length major key and/or elongated major key, being looked into second database
Look for the primary data information；
Conversion subunit, for the primary data information to be converted into the init string.
It is the introduction to the data synchronization unit of database provided in an embodiment of the present invention above, specific implementation can be with
The description in embodiment of the method shown in Fig. 2 is seen above, the effect reached is consistent with above-mentioned embodiment of the method shown in Fig. 2, this
In repeat no more.
In addition, the embodiment of the present invention additionally provides a kind of data synchronizer of database, as shown in fig. 6, the equipment packet
Include processor 601 and memory 602：
The memory 602 is transferred to the processor 601 for storing program code, and by said program code；
The processor 601, for according to any one in the instruction execution embodiment shown in Fig. 2 in said program code
The method of data synchronization of the database of kind embodiment.
In addition, the embodiment of the present invention additionally provides a kind of storage medium, the storage medium is described for storing program code
Program code is used to execute the method for data synchronization of the database of any one embodiment in embodiment shown in Fig. 2.
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.The terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or equipment including a series of elements includes not only those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or equipment institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including institute
State in the process, method, article or equipment of element that there is also other identical elements.
For device embodiments, since it corresponds essentially to embodiment of the method, so related place is referring to method reality
Apply the part explanation of example.System embodiment described above is only schematical, wherein described be used as separating component
The unit of explanation may or may not be physically separated, and the component shown as unit can be or can also
It is not physical unit, you can be located at a place, or may be distributed over multiple network units.It can be according to actual
It needs that some or all of module therein is selected to achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not
In the case of making the creative labor, you can to understand and implement.
The above is only the specific implementation mode of the application, it is noted that for the ordinary skill people of the art
For member, under the premise of not departing from the application principle, several improvements and modifications can also be made, these improvements and modifications are also answered
It is considered as the protection domain of the application.
Claims (11)
1. a kind of method of data synchronization of database, which is characterized in that including：
In response to the triggering command that data synchronize, the log information not synchronized is read from first database；
The log information is parsed to obtain target string；
According to the format of the log information, the target string is converted into target data information；
The target data information is synchronized to the second database.
2. according to the method described in claim 1, it is characterized in that, described parsed to obtain target word to the log information
Symbol string, including：
If the corresponding data manipulation of the log information is insertion operation, searched from the log information fixed-length string and/
Or varying string；
The fixed-length string and/or the varying string are determined as the target string.
3. according to the method described in claim 1, it is characterized in that, described parsed to obtain target word to the log information
Symbol string, including：
It is operated if the corresponding data manipulation of the log information is update, the update is searched from the log information and is operated pair
The more new character strings answered；
Obtain the corresponding init string of primary data information before the update operation；
The init string is updated with the more new character strings, obtains the target string.
4. according to the method described in claim 3, it is characterized in that, it is described obtain it is described update operation before initial number it is believed that
Corresponding init string is ceased, including：
Fixed length major key and/or elongated major key are searched from the log information；
According to the fixed length major key and/or elongated major key, the primary data information is searched in second database；
The primary data information is converted into the init string.
5. method according to any one of claims 1 to 4, which is characterized in that the first database is SQL
Server databases.
6. a kind of data synchronization unit of database, which is characterized in that including：
Reading unit, the triggering command for being synchronized in response to data read the daily record letter not synchronized from first database
Breath；
Resolution unit, for being parsed to obtain target string to the log information；
The target string is converted to target data information by converting unit for the format according to the log information；
Synchronization unit, for the target data information to be synchronized to the second database.
7. device according to claim 6, which is characterized in that the resolution unit includes：
First searches subelement, if being insertion operation for the corresponding data manipulation of the log information, from the log information
Middle lookup fixed-length string and/or varying string；
Determination unit, for the fixed-length string and/or the varying string to be determined as the target string.
8. device according to claim 6, which is characterized in that the resolution unit includes：
Second searches subelement, if being that update operates for the corresponding data manipulation of the log information, from the log information
It is middle to search the corresponding more new character strings of the update operation；
Subelement is obtained, for obtaining the corresponding init string of primary data information before the update operates；
Update subelement obtains the target character for being updated to the init string with the more new character strings
String.
9. device according to claim 8, which is characterized in that the acquisition subelement includes：
Third searches subelement, for searching fixed length major key and/or elongated major key from the log information；
4th searches subelement, for according to the fixed length major key and/or elongated major key, institute to be searched in second database
State primary data information；
Conversion subunit, for the primary data information to be converted into the init string.
10. a kind of data synchronizer of database, the equipment includes processor and memory：
Said program code is transferred to the processor by the memory for storing program code；
The processor is used for according to instruction execution claim 1-5 any one of them databases in said program code
Method of data synchronization.
11. a kind of storage medium, the storage medium is for storing program code, and said program code is for perform claim requirement
The method of data synchronization of 1-5 any one of them databases.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150626.9A CN108363791A (en) | 2018-02-13 | 2018-02-13 | A kind of method of data synchronization and device of database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810150626.9A CN108363791A (en) | 2018-02-13 | 2018-02-13 | A kind of method of data synchronization and device of database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108363791A true CN108363791A (en) | 2018-08-03 |
Family
ID=63002414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810150626.9A Pending CN108363791A (en) | 2018-02-13 | 2018-02-13 | A kind of method of data synchronization and device of database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108363791A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376157A (en) * | 2018-11-21 | 2019-02-22 | 北京像素软件科技股份有限公司 | Data integration method and device |
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN111209344A (en) * | 2020-02-07 | 2020-05-29 | 浪潮软件股份有限公司 | Data synchronization method and device |
| WO2020211248A1 (en) * | 2019-04-19 | 2020-10-22 | 平安科技（深圳）有限公司 | Living body detection log parsing method and apparatus, storage medium and computer device |
| CN112948420A (en) * | 2021-03-05 | 2021-06-11 | 浪潮云信息技术股份公司 | Method for realizing SQL Server increment synchronization |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060255985A1 (en) * | 2005-05-13 | 2006-11-16 | Manisha Agarwala | Reissue an ID to a Data Log Even if the Same ID May Be Repeated |
| US20100036894A1 (en) * | 2008-08-05 | 2010-02-11 | Senda Riro | Data synchronization method, data synchronization program, database server and database system |
| CN102346775A (en) * | 2011-09-26 | 2012-02-08 | 苏州博远容天信息科技有限公司 | Method for synchronizing multiple heterogeneous source databases based on log |
| CN104346479A (en) * | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105183860A (en) * | 2015-09-10 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Data synchronization method and system |
-
2018
- 2018-02-13 CN CN201810150626.9A patent/CN108363791A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060255985A1 (en) * | 2005-05-13 | 2006-11-16 | Manisha Agarwala | Reissue an ID to a Data Log Even if the Same ID May Be Repeated |
| US20100036894A1 (en) * | 2008-08-05 | 2010-02-11 | Senda Riro | Data synchronization method, data synchronization program, database server and database system |
| CN102346775A (en) * | 2011-09-26 | 2012-02-08 | 苏州博远容天信息科技有限公司 | Method for synchronizing multiple heterogeneous source databases based on log |
| CN104346479A (en) * | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105183860A (en) * | 2015-09-10 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Data synchronization method and system |
Non-Patent Citations (1)
| Title |
|---|
| 陈广伟: "面向分布式环境的数据交互与存取模型研究", 《中国优秀硕士学位论文全文数据库 》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109446196A (en) * | 2018-09-26 | 2019-03-08 | 努比亚技术有限公司 | Major key generates processing and determines that method, major key generates and application server |
| CN109376157A (en) * | 2018-11-21 | 2019-02-22 | 北京像素软件科技股份有限公司 | Data integration method and device |
| CN109376157B (en) * | 2018-11-21 | 2021-03-30 | 北京像素软件科技股份有限公司 | Data integration method and device |
| WO2020211248A1 (en) * | 2019-04-19 | 2020-10-22 | 平安科技（深圳）有限公司 | Living body detection log parsing method and apparatus, storage medium and computer device |
| CN111209344A (en) * | 2020-02-07 | 2020-05-29 | 浪潮软件股份有限公司 | Data synchronization method and device |
| CN112948420A (en) * | 2021-03-05 | 2021-06-11 | 浪潮云信息技术股份公司 | Method for realizing SQL Server increment synchronization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108363791A (en) | A kind of method of data synchronization and device of database | |
| TWI719281B (en) | A system, machine readable medium, and machine-implemented method for stream selection | |
| TWI702503B (en) | Systems, methods, and computer readable media to implement merge tree modifications for maintenance operations | |
| US6826726B2 (en) | Remote document updating system using XML and DOM | |
| US8706710B2 (en) | Methods for storing data streams in a distributed environment | |
| CN102122285B (en) | Data cache system and data inquiry method | |
| CA2117846C (en) | Computer method and storage structure for storing and accessing multidimensional data | |
| US9081829B2 (en) | System for organizing and fast searching of massive amounts of data | |
| US8738572B2 (en) | System and method for storing data streams in a distributed environment | |
| US8099421B2 (en) | File system, and method for storing and searching for file by the same | |
| JP5038939B2 (en) | Information retrieval system, method and program | |
| TW201841122A (en) | Key-value store tree | |
| US11487714B2 (en) | Data replication in a data analysis system | |
| US5950191A (en) | Method and system for accessing an item in a linked list using an auxiliary array | |
| KR20160136475A (en) | Managing storage of individually accessible data units | |
| US20190188283A1 (en) | Database Syncing | |
| CN103412803A (en) | Data recovering method and device | |
| CN108491332A (en) | A kind of real-time buffering updating method and system based on Redis | |
| CN107808006B (en) | Fuzzy query method, device and system based on large data volume | |
| CN113139100B (en) | Network flow real-time indexing method and system | |
| US20140201132A1 (en) | Storing a key value to a deleted row based on key range density | |
| CN103390045A (en) | Time sequence storage method and time sequence storage device for monitoring system | |
| CN109408589A (en) | Method of data synchronization and device | |
| CN112328702B (en) | Data synchronization method and system | |
| CN111177144A (en) | Table synchronization method, device, equipment, storage medium and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 110179 No. 177-1 Innovation Road, Hunnan District, Shenyang City, Liaoning Province Applicant after: DongSoft Medical System Co., Ltd. Address before: 110179 No. 177-1 Innovation Road, Hunnan District, Shenyang City, Liaoning Province Applicant before: Dongruan Medical Systems Co., Ltd., Shenyang |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180803 |
|
| RJ01 | Rejection of invention patent application after publication |