CN104951474A - Method and device for acquiring MySQL binlog incremental logs - Google Patents
Method and device for acquiring MySQL binlog incremental logs Download PDFInfo
- Publication number
- CN104951474A CN104951474A CN201410125645.8A CN201410125645A CN104951474A CN 104951474 A CN104951474 A CN 104951474A CN 201410125645 A CN201410125645 A CN 201410125645A CN 104951474 A CN104951474 A CN 104951474A
- Authority
- CN
- China
- Prior art keywords
- mysql
- timestamp
- record
- database
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for acquiring MySQL binlog incremental logs. The method comprises the steps that a first record with a timestamp not earlier than a preassigned timestamp is looked up in a binary log file collection of a current MySQL database; incremental logs of the current MySQL database are acquired according to a filename of a binary log file where the record is located and the location of the record in the file. The invention further discloses a device for acquiring the MySQL binlog incremental logs. By means of the method, start sites of the incremental logs can be acquired through the timestamp in an assigned mode, and therefore the incremental logs can be acquired correctly under the circumstance that a primary MySQL database is changed.
Description
Technical field
The application relates to MySQL database technology, is specifically related to a kind of method for obtaining MySQL binlog Incremental Log.The application provides a kind of device for obtaining MySQL binlog Incremental Log simultaneously.
Background technology
MySQL database has various dissimilar journal file: error log, binary log, inquiry log, slow inquiry log, redo daily record etc., the operation and the information such as state that MySQL database performs can be inquired about according to these daily records, for the management and utilization of MySQL database, these journal files are all indispensable.
Wherein binary log (binlog), have recorded and the change of MySQL database is operated, such as: creation database or table (create), update (insert), renewal rewards theory (update), deletion action (insert) etc., each change operation, with in the form of record write binary log file, all contains the timestamp of current record renewal, this is recorded in the position (that is: side-play amount) in current binary log file and changes to database and operates other relevant information in every bar record.MySQL database has one or more binary log file usually, and different binary log files adopts different numeral number forms to be distinguished by file extension, such as: mysql-bin.00001.Use above-mentioned binary log file, the data recovery function of MySQL database can be realized on the one hand, can copy by active and standby the data consistency realizing MySQL master/slave data storehouse on the other hand.
It is such that MySQL uses binary log file to carry out the active and standby basic process copied: standby database and master data base connect, and acquisition request is from the log recording after the assigned address of appointment binary log file; The log recording that master data base will be arranged in after the assigned address of specifying binary log file, returns to standby database; After standby database receives the data that master data base returns, therefrom parse the change operation performed in master data base, and on standby database, perform these change operations, that is, master data base and standby database perform same database manipulation, thus achieves the data consistency in master/slave data storehouse by active and standby reproduction process.
As can be seen here, the support of MySQL master data base adopts incremental mode, needs the standby database of acquisition binary log data or application program to provide binary log data to other; Accordingly, need to obtain the standby database of binary log data from MySQL master data base or other application programs then need the request obtaining Incremental Log data, specify clearly and wish therefrom to obtain the filename of the binary log file of daily record data and the side-play amount of the daily record data of required acquisition in described binary log file, the binary log data needed for can correctly obtaining from master data base like this.
Above by the mode of specifying " filename+side-play amount " to obtain Incremental Log, contribute to reducing the interaction data amount between master data base and the application program obtaining binary log data for database or other, reduce taking bandwidth, but there is certain defect in specific implementation process.Capture increment binary log data for needing from master data base, to meet the application program of the related service demands such as increment subscription or consumption, can adopt under normal circumstances and capture increment binary log data with mode like standby class database from master data base.But when master data base fault causes master/slave data storehouse to switch, because the binary log file of master data base and standby database is numbering independent of one another, corresponding relation is there is not between active and standby, and the initiation site information (filename+side-play amount) that the application program capturing Incremental Log uses and master data base are associated, therefore the initiation site corresponding with former initiation site information cannot be found in the binary log file of standby database (current master data base), thus cause described application program cannot continue the Incremental Log data capturing Incremental Log data or grab mistake.
Summary of the invention
The application provides a kind of method for obtaining MySQL binlog Incremental Log, adopts " filename+side-play amount " to specify the mode in site correctly cannot obtain the problem of Incremental Log when MySQL master data base changes to solve prior art.The application provides a kind of device for obtaining MySQL binlog Incremental Log in addition.
The application provides a kind of method for obtaining MySQL binlog Incremental Log, comprising:
In the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp;
Be recorded in the position in described file according to the filename of this record place binary log file and this, obtain the Incremental Log of described current MySQL database.
Optionally, described in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, comprising:
In the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp;
First record that its timestamp is no earlier than described preassigned timestamp is searched in last file described.
Optionally, perform in the binary log file set of described current MySQL database, when the timestamp searching first record that it comprises is not later than the step of last file of described preassigned timestamp, employing be binary chop.
Optionally, described method comprises:
Data cube computation is set up with described current MySQL database;
Obtain the binary log listed files of described current MySQL database;
Accordingly, described in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, the search operation performed in the file set comprised at the binary log listed files of above-mentioned acquisition exactly.
Optionally, described and described current MySQL database is set up data cube computation and is comprised:
Obtain the port numbers that the IP address of described current MySQL database place server and described current MySQL database provide database service;
Described IP address and described port numbers and described current MySQL database is used to set up data cube computation.
Optionally, the IP address of described acquisition described current MySQL database place server and described current MySQL database provide the port numbers of database service, realize in the following way:
Read information relevant to described current MySQL database in metamessage storehouse, and therefrom extract described IP address and described port numbers.
Optionally, described method comprises:
Record the timestamp of last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
Optionally, at last timestamp recorded of the Incremental Log that described record obtains from MySQL master data base, before the step as described preassigned timestamp, following step is performed:
Judge whether MySQL database system performs active-standby switch operation;
If so, described MySQL master data base will be taken over continue to provide the MySQL standby database of database service, as described current MySQL database.
Optionally, described judge MySQL database system whether perform active-standby switch operation, in the following way realize:
Monitoring be stored in metamessage storehouse, whether change for the field of the IP address of recording MySQL master data base place server; If so, then judge that MySQL database system performs active-standby switch operation.
The application also provides a kind of device for obtaining MySQL binlog Incremental Log, comprising:
Record search unit, in the binary log file set of current MySQL database, searches first record that its timestamp is no earlier than preassigned timestamp;
Incremental Log acquiring unit, is recorded in the position in described file, obtains the Incremental Log of described current MySQL database for the filename of record place binary log file that finds according to described record search unit and this.
Optionally, described record search unit comprises:
Ff subelement, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp;
Record search performs subelement, is no earlier than first record of described preassigned timestamp for searching its timestamp in last file described.
Optionally, described ff subelement specifically for, adopt binary chop, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp.
Optionally, described device comprises:
Connection establishment unit, for setting up data cube computation with described current MySQL database;
Listed files acquiring unit, for obtaining the binary log listed files of described current MySQL database;
Accordingly, search operation is performed in the file set that described record search unit comprises at the binary log listed files that described listed files acquiring unit exports.
Optionally, described connection establishment unit comprises:
Link information obtains subelement, and IP address and described current MySQL database for obtaining described current MySQL database place server provide the port numbers of database service;
Connection establishment performs subelement, sets up data cube computation for using the IP address of described link information acquisition subelement output and port numbers and described current MySQL database.
Optionally, described link information obtain subelement specifically for, read information relevant to described current MySQL database in metamessage storehouse, and therefrom extract described IP address and described port numbers.
Optionally, described device comprises:
Timestamp record cell, for recording the timestamp of last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
Optionally, described device comprises:
Active-standby switch judging unit, for judging whether MySQL database system performs active-standby switch operation;
Current database selection unit, for when the output of described active-standby switch judging unit is "Yes", will take over described MySQL master data base and continue to provide the MySQL standby database of database service, as described current MySQL database.
Optionally, described active-standby switch judging unit specifically for, monitoring be stored in metamessage storehouse, whether change for the field of the IP address of recording MySQL master data base place server; If so, then judge that MySQL database system performs active-standby switch operation.
Compared with prior art, the application has the following advantages:
The method for obtaining MySQL binlog Incremental Log of the application, do not adopt prior art according to the mode of filename and side-play amount determination initiation site, but the mode that have employed according to timestamp location initiation site, that is: in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, and the position be recorded according to the filename of this record place binary log file and this in described file, obtain the Incremental Log data of described current MySQL database, achieve time-based Incremental Log acquisition methods, thus still can correctly obtain Incremental Log data when MySQL database changes.
Accompanying drawing explanation
Fig. 1 is a kind of embodiment of the method process flow diagram for obtaining MySQL binlog Incremental Log of the application;
Fig. 2 is the embodiment process flow diagram that the timestamp of employing binary search its comprise first record of the application is not later than last file of described preassigned timestamp;
Fig. 3 is the embodiment process flow diagram of the processing procedure needed under active-standby switch scene of the application;
Fig. 4 is a kind of device embodiment schematic diagram for obtaining MySQL binlog Incremental Log of the application.
Embodiment
Set forth a lot of detail in the following description so that fully understand the application.But the application can be much different from alternate manner described here to implement, those skilled in the art can when doing similar popularization without prejudice to when the application's intension, and therefore the application is by the restriction of following public concrete enforcement.
In this application, a kind of method for obtaining MySQL binlog Incremental Log and a kind of device for obtaining MySQL binlog Incremental Log is each provided.For the ease of understanding the technical scheme of the application, before detailed description technical scheme, first brief description is done to binlog file layout.
Binlog is the daily record that MySQL stores in binary form, and each binlog top of file is the mark of 4 bytes, be worth for 0xfe0x620x690x6e(namely: add bin after 0xfe).Be exactly a series of LOG_EVENT(log event after file header), LOG_EVENT is the base unit in binlog file, and namely binlog increases according to the form by LOG_EVENT under normal circumstances.Each LOG_EVENT is separate unit, the relation do not quoted mutually.The event type of LOG_EVENT has a variety of, and the event type wherein for the LOG_EVENT of database of record change operation is query event (queryevent).LOG_EVENT in above-mentioned binlog file structure is exactly the record described in the application.
Each LOG_EVENT has a length to be the event header of 19 bytes, wherein comprises the information such as timestamp, event type of the length of event, event.Be exactly according to event type definite event body after event header,
For the record that type is query event, in event body, have recorded agent thread ID, error code, concrete database change action statement etc.Binlog file layout is as shown in the table:
Table one: binlog file layout (wherein N byte all refers to the sequence from 0)
Wherein, the host time of the execution event of the 0-3 byte records of LOG_EVENT head, be the timestamp information described in the application, namely current record upgrade time, such as: timestamp 1390467118 correspondence be exactly this time of Thu Jan2316:51:58CST2014.The 13-16 byte of LOG_EVENT head, is the side-play amount (record is arranged in the position of binary log file) described in the application.
On the basis understanding binlog file layout, below the embodiment of technical scheme is described in detail one by one.Please refer to Fig. 1, it is a kind of embodiment of the method process flow diagram for obtaining MySQL binlog Incremental Log of the application.Described method comprises the steps:
Step 101: in the binary log file set of current MySQL database, searches first record that its timestamp is no earlier than preassigned timestamp.
The technical scheme of the application, according to preassigned timestamp, in the binary log file set of the current MySQL database of Incremental Log to be obtained, locate the initiation site corresponding with described timestamp, that is: first record that its timestamp is no earlier than described timestamp is searched, and be recorded in side-play amount in described file as initiation site using the filename of this record place file and this, obtain the Incremental Log data of current MySQL database.
Can be found out by description above, the core of technical scheme is: locate initiation site by timestamp information, and obtains the Incremental Log data of MySQL database according to initiation site.Therefore, the technical scheme of the application both can be deployed in the server end at described MySQL database place, the local mode read is taked to obtain Incremental Log data, also can be deployed in be different from MySQL database place server other equipment on, and by set up network connect mode obtain Incremental Log data.These two kinds of embodiments do not depart from the core of the application.
Technical scheme, according to the function of timestamp location initiation site, is realized the search procedure of the binary log file set of described current MySQL database by this step.First introduction concise and to the point to the binary log file set cooperation described in this step below.
MySQL database can create the new file of two similar xxx-bin.000001 and xxx-bin.index of title at data root directory when initial launch, if the relevant options of configuration file does not provide filename, use Hostname is named these two files by MySQL, such as: mysql-bin.000001 and mysql-bin.index.Wherein, MySQL can charge to mysql-bin.000001 file all amendment situations to database content and structure, and mysql-bin.index file then contains the inventory of all binary log files of a current use.MySQL database can when restarting or log file size exceed that system variable max_binlog_size specifies upper in limited time, generate a new journal file, its file extension sequence number increases progressively successively.As can be seen here, the binary log file of MySQL database may be one, may be also one group, and the binary log file set described in the application then contains all binary log files of the current use of MySQL database.
Due to the change operation that binary log file is done MySQL database according to sequence of event, and binary log file set is also by tactic (numeral in file extension successively add up) of morning to evening usually according to the binary log file generated, therefore, search operation described in this step refers to, comprise at the binary log file set of described current MySQL database, search operation is performed by the binary log file early sorted to evening according to the time, search first record that its timestamp is no earlier than preassigned timestamp.
By can know the introduction of binlog file layout above, binlog file just as a stream file, the change that MySQL database is done according to sequence of event.And its each record wherein does not significantly start and end mark, can only judge an end position recorded by length.Based on this feature of binlog file layout, will from the binary log file set of described current MySQL database, its timestamp be found to be no earlier than described preassigned timestamp, the most simple method, be exactly analyze each binary log file one by one, and when analyzing each file, from its head, all read each record and judge whether the timestamp of current record is no earlier than described preassigned timestamp, if, then have found required record, otherwise, continue to read next record.Adopt in this way, if the record of required location is positioned at the afterbody of last binary log file, so aforesaid operations is exactly the traversal to whole binary log file, and the execution efficiency of whole search procedure is very low.
In order to improve above-mentioned search efficiency, the technical scheme of the application provides the binary log file first determining record place to be found, and then in described binary log file, search the method for described record, thus the traversing operation (only needing to travel through a binary log file) to binary log file can be reduced.Respectively locating file and these two steps of position the record are elaborated below.
Step 101-1: in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp.
The concrete search procedure of this step is such, read first record in each binary log file, judge that whether its timestamp is early than described preassigned timestamp, if, then read first record of next binary log file, and do same judgement, until find first of certain file timestamp recorded to be later than described preassigned timestamp, then the file searched before current file is the binary log file of this step desired seek.If the timestamp of first of current file record is early than described preassigned timestamp, and follow-uply there is no other binary log files (current file is last file in described set), illustrate that record to be found is arranged in current file, current file is exactly the binary log file of this step desired seek.
It should be noted that, in the process above by the binary log file needed for Search and Orientation, if the timestamp of first record in the binary log file of current reading equals described preassigned timestamp just, illustrate and have found required record, so this search procedure just can stop, nor subsequent step 101-2(must be performed again in the above-mentioned file found, search first record that timestamp is no earlier than described preassigned timestamp further), step 101 just all completes.
In above-described embodiment, if the binary log file of desired seek is last file in described binary log file set, so above-mentioned Search and Orientation process just needs to read first record for each file, line time of going forward side by side stamp judges, under the application scenarios of the MySQL database of reality, file in binary log file set may have tens, tens even more, in this case, adopt above-mentioned reading one by one and the mode judged, treatment effeciency is still relatively low.In order to effectively improve this problem, the technical scheme of the application provides a kind of preferred implementation, that is: adopt binary chop, and the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp.
Binary chop, also known as binary search, it is the lookup method that a kind of efficiency is higher, its basic thought is as follows: tentation data sequence is by ascending sort (descending sort reason is similar), for set-point x, compare from the centre position (as current location) of sequence, if present position values equals x, then search successfully; If x is less than present position values, then in the first half section of data sequence, continue above-mentioned search procedure; If x is greater than present position values, in the second half section of data sequence, continue above-mentioned search procedure, until find.Due to binary chop, have employed the mode of inquiry by half, exclude the half of data sequence to be checked at every turn, therefore can reduce the number of times searched.Certain number is searched for comprising in the data sequence of N number of number at one, the conventional time complexity comparing lookup algorithm is one by one O (N), if but this data sequence is orderly, the time complexity of binary chop is so adopted to be only O (log2N) (wherein 2 is truth of a matter of logarithm).Such as: for the sequence comprising 7 data, adopt the conventional algorithm compared one by one to need at most to compare 7 times, just can complete, and adopt binary chop, at most only need to compare 3 times and just can complete.
Specific to the technical scheme of the application, because the record in binary log file writes according to the time sequencing performing Relational database change operation, and binary log file also generates (its extension name increases progressively) one by one according to time sequencing, therefore also arrange according to time sequencing according to the timestamp of first record in whole binary log files of time sequencing arrangement, meeting binary chop requirement data sequence must be this orderly basic demand, therefore the binary log file needed for this step of binary search can be adopted, thus effectively improve the efficiency of searching.Refer to accompanying drawing 2, the timestamp of its first record comprised of its employing binary search being the application is not later than the basic procedure schematic diagram of last file of described preassigned timestamp, due to binary chop comparative maturity, therefore below only (for convenience of description, be " be not later than t1 " by search criterion " timestamp of its first record comprised is not later than described preassigned timestamp " simplified characterization) is briefly described in conjunction with this accompanying drawing for how adopting binary chop in the technical program:
Step 101-1-1: be algorithm correlated variables initialize.Current have N number of binary log file, and with array B representative, B [0] is first file, and B [N-1] is last file.L represents the start position in interval to be found, and h represents the end position in interval to be found, and i represents the centre position in interval to be found.Preassigned timestamp is t1.
Step 101-1-2: the centre position of the paper interval of location current lookup, and the timestamp t obtaining first record of respective file.
Step 101-1-3: if t=t1, then illustrate and have found required record (first record in B [i] file), therefore this search procedure terminates.
Step 101-1-4: judge whether l is less than or equal to h, if illustrate that search procedure not yet terminates, continues to perform follow-up search operation.
Step 101-1-5: if t<t1, that is: the timestamp of current record is early than t1, but B [i] file not necessarily meets last file that described " being not later than t1 " requires, therefore also need to continue to search in the second half section in current file interval, look at that but the timestamp that whether there is its first record is later than the file of this logging timestamp early than t1, forward step 101-1-6 to and perform.
Step 101-1-6: arrange l=i+1 is the start position continuing to search between setting area in the second half section in current file interval, forwards step 101-1-2 to and performs.
Step 101-1-7: if t>t1, that is: the timestamp of current record is later than t1, and B [i] file does not meet the requirement of " being not later than t1 ", should continue to search in the first half section in current file interval, forwards step 101-1-8 to and performs.
Step 101-1-8: arrange h=i-1 is the end position continuing to search between setting area in the first half section in current file interval, forwards step 101-1-2 to and performs.
Above-mentioned search procedure circulates execution always, this process stops in both cases: the first situation, t=t1 is judged in step 101-1-3, then B [i] is exactly last file met the demands, and first record in B [i] is exactly the record of the technical scheme desired seek of the application, in this case can skip subsequent step 101-2, directly perform step 102; The second situation, judges that in step 101-1-4 l>h(reference position is greater than end position), illustrate that binary chop process terminates, current B [i] file meets last file that " being not later than t1 " requires exactly.
Step 101-2: search first record that its timestamp is no earlier than described preassigned timestamp in last file described.
Perform this step, illustrate in the binary log file set of described current MySQL database, the timestamp that have found first record that it comprises is not later than last file of described preassigned timestamp, only need in this step from first record of last file described, travel through all records in this file, find its timestamp to be no earlier than first record of described preassigned timestamp.
Here it should be noted that, under normal circumstances, its timestamp can be found in last file described to equal first record of described preassigned timestamp, why have employed the form of presentation of " being no earlier than " (comprise and be equal to and greater than two kinds of situations) in this step, consider contingent special circumstances in specific implementation process: in the binary log file set of described current MySQL database, its timestamp and the identical record of described preassigned timestamp (detailed description refers to hereinafter relevant with active-standby switch explanation) cannot be found.In this case, still initiation site as far as possible accurately can be found in order to make technical scheme, and ensure that the step 102 of follow-up acquisition Incremental Log can perform smoothly, therefore have employed in this step search its timestamp be no earlier than described in advance the fixed time stamp first record mode.
Such as, in an object lesson of the present embodiment, be achieved in that from first record of last file described, if the timestamp of this record is less than preassigned timestamp, then continue to read next record, and do same judgement ... until the timestamp of current record is greater than or equal to described preassigned timestamp, then this search procedure terminates, first record that current record is exactly in last file described, its timestamp is no earlier than described preassigned timestamp.
So far, just in the binary log file set of described current MySQL database, have found corresponding record according to described preassigned timestamp, so, just can perform according to the record found the function that subsequent step 102 realizes obtaining Incremental Log.
Consider in actual applications, need from MySQL database capture increment binary log data, with meet increment subscribe to or the application program of the related service demands such as consumption or system be usually arranged on be different from MySQL database place server other equipment, connected by network in normal conditions and capture Incremental Log data from MySQL master data base, under the scene that active-standby switch occurs, the method then adopting the application to provide, continues to capture Incremental Log data from standby database (that is: the current MySQL database described in the application).In this case, from last timestamp recorded of the Incremental Log that MySQL master data base obtains, the preassigned timestamp described in this step is.In order to implement the technical scheme of the application under above-mentioned application scenarios, execution before execution this step 101 is then needed to judge active-standby switch, logging timestamp, set up data cube computation, obtain these four steps of binary log listed files, refer to accompanying drawing 3, it is the embodiment process flow diagram of above-mentioned processing procedure, is further detailed below in conjunction with accompanying drawing 3.For the ease of following description, will the application program or the system that achieve technical scheme in Incremental Log process captured, be referred to as Incremental Log and capture task.
Step 100-1: judge whether MySQL database system performs active-standby switch operation.
Incremental Log captures task and adopts the mode similar with MySQL standby database, according to the requirement of MySQL network communication protocol, to the Packet Generation of COM_BINLOG_DUMP order be encapsulated to MySQL master data base, filename and side-play amount that hope therefrom obtains the binary log file of Incremental Log data also will be encapsulated in this packet, after MySQL master data base receives this packet, satisfactory Incremental Log can be returned to described Incremental Log and capture task.
Consider that MySQL master data base may break down, in order in time the operation capturing Incremental Log is switched to (that is: the master data base after switching) on standby database, Incremental Log captures task needs monitoring MySQL database system whether to there occurs active-standby switch.In an object lesson of the present embodiment, current master/slave data storehouse is characterized with a specific character string " GroupKey " in the metamessage storehouse of database, by detecting this character string, just can obtain the relevant information of current master data base and standby database, if find that the IP address of master data base there occurs change, namely can judge that master/slave data storehouse there occurs switching, described Incremental Log captures task just to be needed by performing follow-up step, the operation capturing Incremental Log is switched on standby database.
Step 100-2: the timestamp recording last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
In order to the operation obtaining binlog Incremental Log from MySQL master data base is switched to (that is: the current MySQL database described in the application) on MySQL standby database, need the timestamp of last record of the Incremental Log obtained from MySQL master data base to record, the follow-up operation of searching initiation site according to described timestamp in the binary log file set of standby database could be performed like this.
In this step, the timestamp of last record of the Incremental Log obtained from MySQL master data base is recorded, the namely timestamp information of the 0-3 byte of the head of record last record described, as follow-up foundation of locating initiation site in the binary log file set of standby database, thus ensure that the Incremental Log obtained from standby database is continuous print with the Incremental Log obtained from master data base as far as possible, avoid the phenomenon occurring loss of data as far as possible.The timestamp that this step obtains, is the preassigned timestamp described in the application.
Step 100-3: set up data cube computation with described current MySQL database.
Incremental Log crawl task follows the interaction protocol between MySQL master/slave data storehouse, with described current MySQL database (after there is active-standby switch, namely standby database originally becomes the current MySQL database described in the application) set up data cube computation, that is Incremental Log captures task with the identity of standby database to described current MySQL database request Incremental Log data.
Data cube computation is set up with described current MySQL database described in this step, have employed relatively wide in range form of presentation, in concrete enforcement, this step not only comprises establishment of connection process, usually further comprises the processes such as authentication, its basic procedure is such: first need the IP address of acquisition described current MySQL database place server and described current MySQL database to provide the port numbers of database service, then a socket(socket is created) set up data cube computation with described current MySQL database, and perform according to the interaction protocol between MySQL master/slave data storehouse and shake hands and the reciprocal process such as checking, in this process, described Incremental Log captures task may also be needed to provide user name according to the requirement of MySQL master server, the information such as password, and only have the authentication by described current MySQL database, described data cube computation is just successfully established.
In order to complete above-mentioned connection establishment process, the IP address of current MySQL database must be obtained and the port numbers of database service is provided.In an object lesson of the present embodiment, current master/slave data storehouse is characterized with a specific character string " GroupKey " in the metamessage storehouse of database, by detecting this character string, just can obtain the relevant information of current master data base and standby database, after generation active-standby switch, standby database originally becomes current master data base (the current MySQL database described in the application), therefore the information relevant to current MySQL database be can obtain from above-mentioned character string, and described IP address and described port numbers therefrom extracted.In other embodiments, other obtain manners can be adopted, such as: when not revising the port in MySQL configuration file and arranging, the port numbers 3306 that can directly use MySQL to give tacit consent to is to set up data cube computation.
Step 100-4: the binary log listed files obtaining described current MySQL database.
In order to realize the technical scheme of the application, according to the timestamp information obtained in step 100-2, the site corresponding with this timestamp is searched in the binary log file set of current MySQL database, also need the binary log listed files obtaining described current MySQL database, that is: determine the seek scope of the step 101 of technical scheme.
In an object lesson of the present embodiment, obtain binary log listed files in the following way: Incremental Log captures task based on the data cube computation set up in step 100-3, " show master logs " order is sent to described current MySQL database, after described current MySQL database receives this order, the binary log listed files of its this locality will be returned to Incremental Log and capture task, such as, in this example, the binary log listed files that Incremental Log crawl task receives comprises 7 files, mysql-bin.000001, mysql-bin.000002, ... mysql-bin.000007.
In other embodiments, also other modes being different from above-mentioned object lesson can be adopted to obtain the binary log listed files of described current MySQL database, such as can be accessed the binary log index file (such as: mysql-bin.index) of described current MySQL database by the mode of file-sharing, therefrom read required listed files information.
So far, the task that captures Incremental Log just can continue to perform step 101, search the record corresponding with this timestamp in the file set comprised at the above-mentioned binary log listed files obtained according to the timestamp obtained, that is: search first record that its timestamp is no earlier than preassigned timestamp.
Mentioned in a step 101 above, why to search first record that its timestamp is no earlier than preassigned timestamp, instead of directly search first record that its timestamp equals preassigned timestamp, the special circumstances considered in specific implementation, these special circumstances are relevant to above-described active-standby switch scene, are described in further detail below to this.
When MySQL master data base breaks down (machine of such as delaying), there is certain delay in the active and standby reproduction process due to MySQL, therefore there is the standby database up-to-date Incremental Log before machine of not yet master data base being delayed and pull the situation of this locality, at this moment there occurs active-standby switch, standby database becomes master data base, continue to provide database service, continue to write new record in binary log file.In this case, standby database lost master data base and to delay the Partial Incremental log recording before machine, and Incremental Log captures task and likely successfully captured this part record.Therefore, when Incremental Log captures the timestamp of task according to last record in this part record, when searching in the binary log file set of standby database (that is: the current MySQL database described in the application), just cannot find its timestamp and the identical record of this timestamp.Still site as far as possible accurately can be found in this case in order to make technical scheme, and ensure that the task of capturing Incremental Log can perform smoothly, what therefore search in a step 101 is first record that its timestamp is no earlier than described preassigned timestamp.
It should be explicitly made clear at this point, the technical scheme of the application is not restricted to the application scenarios of above-mentioned generation active-standby switch, also can be applied in other application scenarioss, and above mentioned step is not also all required.Such as: achieve this locality (on same station server) that the application program of technical scheme or system are positioned at current MySQL database, and need to obtain Incremental Log according to the temporal information of specifying, so do not need to perform above-mentioned to active-standby switch with set up the relevant sub-step of data cube computation (step 100-1 is to step 100-4), but directly adopt the mode of traversal or adopt the first locating file (step 101-1) in the present embodiment, relocate the mode of record (step 101-2), that carries out recording in the binary log file set of this locality searches.Which kind of mode no matter is adopted to realize this step; as long as its timestamp can be found to be no earlier than first of preassigned timestamp record; the change of above-mentioned embodiment, does not depart from the core of technical scheme, within the protection domain of the application yet.
Step 102: be recorded in the position in described file according to filename and this of this record place binary log file, obtain the Incremental Log of described current MySQL database.
By step 101, in the binary log file set of current MySQL database, have found the record corresponding with preassigned timestamp, according to the form of the binlog file introduced above, the 13-16 byte reading this record head just obtains this and is recorded in side-play amount in current binary log file, by filename and the described side-play amount of this record place binary log file, just obtain the site information of described preassigned timestamp in described MySQL binary log file set, current MySQL database just can be obtained further according to this site information, be positioned at the Incremental Log data after this site information.
In specific implementation process, can realize by sending to current MySQL database the order obtaining Incremental Log data.Task is captured for the Incremental Log described in the present embodiment, after MySQL database system generation active-standby switch, find the position that side-play amount that the record corresponding with preassigned timestamp be positioned at mysql-bin.000003 file is 126 bytes in a step 101.Incremental Log captures the data cube computation of task by having set up with current MySQL database, the native command COM_BINLOG_DUMP of MySQL is sent to current MySQL database, and according to the call format of this order, carry the site information wishing to obtain Incremental Log data, that is: file is called mysql-bin.000003, and side-play amount is 126 bytes.After described current MySQL server receives above-mentioned COM_BINLOG_DUMP order, the site information that can carry according to this order, captures task to Incremental Log and pushes the Incremental Log data be positioned at after this site information.So far, after MySQL database system generation active-standby switch, Incremental Log crawl task also achieves the handoff procedure from master data base to standby database, thus ensure that the continuity and correctness that capture Incremental Log data to greatest extent.Incremental Log captures the Incremental Log data that task then can resolve acquisition further, and carries out necessary subsequent treatment according to relevant business demand.
The mode of acquisition Incremental Log presented hereinbefore, captures task for background with Incremental Log, and which needs the mode (being connected with Database) by network service to obtain the Incremental Log data of current MySQL database.But in other implementations, the technical scheme of the application also by network, directly can not perform the operation obtaining Incremental Log data in this locality of MySQL database.Such as, fopen(is used in this locality of described current MySQL database) function or similar function open the binary log file that hope therefrom obtains Incremental Log data, and by seek () function or similar function by file read-write needle locating to the offset location of wishing acquisition Incremental Log data, then call fread () or similar function read according to binary log file layout from the side-play amount place specified needed for data, the function of the described current MySQL database Incremental Log data of acquisition described in this step can be realized equally.
In sum, no matter be adopt local mode, or adopt the mode by network, also no matter the trigger event of technical scheme is active-standby switch event described in the present embodiment or other events, as long as need to determine site (filename+side-play amount) according to temporal information in the binary log file set of MySQL, and obtain the application scenario of MySQL Incremental Log data based on this site, the technical scheme of the application can be adopted.
The method for obtaining MySQL binlog Incremental Log of the application, do not adopt prior art according to the mode of filename and side-play amount determination initiation site, but the mode that have employed according to timestamp location initiation site, that is: in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, and the position be recorded according to the filename of this record place binary log file and this in described file, obtain the Incremental Log data of described current MySQL database, achieve time-based Incremental Log acquisition methods, thus still can correctly obtain Incremental Log data when MySQL database changes.
In the above-described embodiment, providing a kind of method for obtaining MySQL binlog Incremental Log, corresponding, the application also provides a kind of device for obtaining MySQL binlog Incremental Log.
Please refer to Fig. 4, its a kind of device embodiment schematic diagram for obtaining MySQL binlog Incremental Log provided for the application.Because device embodiment is substantially similar to embodiment of the method, so describe fairly simple, the correspondence that relevant part refers to embodiment of the method illustrates.The device embodiment of following description is only schematic.
A kind of device for obtaining MySQL binlog Incremental Log of the present embodiment, comprise: record search unit 401, for in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp; Incremental Log acquiring unit 402, is recorded in the position in described file, obtains the Incremental Log of described current MySQL database for the filename of record place binary log file that finds according to described record search unit and this.
Optionally, described record search unit comprises:
Ff subelement, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp;
Record search performs subelement, is no earlier than first record of described preassigned timestamp for searching its timestamp in last file described.
Optionally, described ff subelement specifically for, adopt binary chop, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp.
Optionally, described device comprises:
Connection establishment unit, for setting up data cube computation with described current MySQL database;
Listed files acquiring unit, for obtaining the binary log listed files of described current MySQL database;
Accordingly, search operation is performed in the file set that described record search unit comprises at the binary log listed files that described listed files acquiring unit exports.
Optionally, described connection establishment unit comprises:
Link information obtains subelement, and IP address and described current MySQL database for obtaining described current MySQL database place server provide the port numbers of database service;
Connection establishment performs subelement, sets up data cube computation for using the IP address of described link information acquisition subelement output and port numbers and described current MySQL database.
Optionally, described link information obtain subelement specifically for, read information relevant to described current MySQL database in metamessage storehouse, and therefrom extract described IP address and described port numbers.
Optionally, described device comprises:
Timestamp record cell, for recording the timestamp of last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
Optionally, described device comprises:
Active-standby switch judging unit, for judging whether MySQL database system performs active-standby switch operation;
Current database selection unit, for when the output of described active-standby switch judging unit is "Yes", will take over described MySQL master data base and continue to provide the MySQL standby database of database service, as described current MySQL database.
Optionally, described active-standby switch judging unit specifically for, monitoring be stored in metamessage storehouse, whether change for the field of the IP address of recording MySQL master data base place server; If so, then judge that MySQL database system performs active-standby switch operation.
Although the application with preferred embodiment openly as above; but it is not for limiting the application; any those skilled in the art are not departing from the spirit and scope of the application; can make possible variation and amendment, the scope that therefore protection domain of the application should define with the application's claim is as the criterion.
In one typically configuration, computing equipment comprises one or more processor (CPU), input/output interface, network interface and internal memory.
Internal memory may comprise the volatile memory in computer-readable medium, and the forms such as random access memory (RAM) and/or Nonvolatile memory, as ROM (read-only memory) (ROM) or flash memory (flash RAM).Internal memory is the example of computer-readable medium.
1, computer-readable medium comprises permanent and impermanency, removable and non-removable media can be stored to realize information by any method or technology.Information can be computer-readable instruction, data structure, the module of program or other data.The example of the storage medium of computing machine comprises, but be not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic RAM (DRAM), the random access memory (RAM) of other types, ROM (read-only memory) (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc ROM (read-only memory) (CD-ROM), digital versatile disc (DVD) or other optical memory, magnetic magnetic tape cassette, tape magnetic rigid disk stores or other magnetic storage apparatus or any other non-transmitting medium, can be used for storing the information can accessed by computing equipment.According to defining herein, computer-readable medium does not comprise non-temporary computer readable media (transitory media), as data-signal and the carrier wave of modulation.
2, it will be understood by those skilled in the art that the embodiment of the application can be provided as method, system or computer program.Therefore, the application can adopt the form of complete hardware embodiment, completely software implementation or the embodiment in conjunction with software and hardware aspect.And the application can adopt in one or more form wherein including the upper computer program implemented of computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) of computer usable program code.
Claims (18)
1. for obtaining a method for MySQL binlog Incremental Log, it is characterized in that, comprising:
In the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp;
Be recorded in the position in described file according to the filename of this record place binary log file and this, obtain the Incremental Log of described current MySQL database.
2. the method for obtaining MySQL binlog Incremental Log according to claim 1, it is characterized in that, described in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, comprising:
In the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp;
First record that its timestamp is no earlier than described preassigned timestamp is searched in last file described.
3. the method for obtaining MySQL binlog Incremental Log according to claim 2, it is characterized in that, perform in the binary log file set of described current MySQL database, when the timestamp searching first record that it comprises is not later than the step of last file of described preassigned timestamp, employing be binary chop.
4. the method for obtaining MySQL binlog Incremental Log according to claim 1-3 any one, is characterized in that, comprising:
Data cube computation is set up with described current MySQL database;
Obtain the binary log listed files of described current MySQL database;
Accordingly, described in the binary log file set of current MySQL database, search first record that its timestamp is no earlier than preassigned timestamp, the search operation performed in the file set comprised at the binary log listed files of above-mentioned acquisition exactly.
5. the method for obtaining MySQL binlog Incremental Log according to claim 4, is characterized in that, described and described current MySQL database is set up data cube computation and comprised:
Obtain the port numbers that the IP address of described current MySQL database place server and described current MySQL database provide database service;
Described IP address and described port numbers and described current MySQL database is used to set up data cube computation.
6. the method for obtaining MySQL binlog Incremental Log according to claim 5, it is characterized in that, the IP address of described acquisition described current MySQL database place server and described current MySQL database provide the port numbers of database service, realize in the following way:
Read information relevant to described current MySQL database in metamessage storehouse, and therefrom extract described IP address and described port numbers.
7. the method for obtaining MySQL binlog Incremental Log according to claim 4, is characterized in that, comprising:
Record the timestamp of last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
8. the method for obtaining MySQL binlog Incremental Log according to claim 7, it is characterized in that, at last timestamp recorded of the Incremental Log that described record obtains from MySQL master data base, before the step as described preassigned timestamp, perform following step:
Judge whether MySQL database system performs active-standby switch operation;
If so, described MySQL master data base will be taken over continue to provide the MySQL standby database of database service, as described current MySQL database.
9. the method for obtaining MySQL binlog Incremental Log according to claim 8, is characterized in that, described judge MySQL database system whether perform active-standby switch operation, in the following way realize:
Monitoring be stored in metamessage storehouse, whether change for the field of the IP address of recording MySQL master data base place server; If so, then judge that MySQL database system performs active-standby switch operation.
10. for obtaining a device for MySQL binlog Incremental Log, it is characterized in that, comprising:
Record search unit, in the binary log file set of current MySQL database, searches first record that its timestamp is no earlier than preassigned timestamp;
Incremental Log acquiring unit, is recorded in the position in described file, obtains the Incremental Log of described current MySQL database for the filename of record place binary log file that finds according to described record search unit and this.
11. devices for obtaining MySQL binlog Incremental Log according to claim 10, is characterized in that, described record search unit comprises:
Ff subelement, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp;
Record search performs subelement, is no earlier than first record of described preassigned timestamp for searching its timestamp in last file described.
12. devices for obtaining MySQL binlog Incremental Log according to claim 11, it is characterized in that, described ff subelement specifically for, adopt binary chop, in the binary log file set of described current MySQL database, the timestamp searching first record that it comprises is not later than last file of described preassigned timestamp.
13. devices for obtaining MySQL binlog Incremental Log according to claim 10-12 any one, is characterized in that, comprising:
Connection establishment unit, for setting up data cube computation with described current MySQL database;
Listed files acquiring unit, for obtaining the binary log listed files of described current MySQL database;
Accordingly, search operation is performed in the file set that described record search unit comprises at the binary log listed files that described listed files acquiring unit exports.
14. devices for obtaining MySQL binlog Incremental Log according to claim 13, is characterized in that, described connection establishment unit comprises:
Link information obtains subelement, and IP address and described current MySQL database for obtaining described current MySQL database place server provide the port numbers of database service;
Connection establishment performs subelement, sets up data cube computation for using the IP address of described link information acquisition subelement output and port numbers and described current MySQL database.
15. devices for obtaining MySQL binlog Incremental Log according to claim 14, it is characterized in that, described link information obtain subelement specifically for, read information relevant to described current MySQL database in metamessage storehouse, and therefrom extract described IP address and described port numbers.
16. devices for obtaining MySQL binlog Incremental Log according to claim 13, is characterized in that, comprising:
Timestamp record cell, for recording the timestamp of last record of the Incremental Log obtained from MySQL master data base, as described preassigned timestamp.
17. devices for obtaining MySQL binlog Incremental Log according to claim 16, is characterized in that, comprising:
Active-standby switch judging unit, for judging whether MySQL database system performs active-standby switch operation;
Current database selection unit, for when the output of described active-standby switch judging unit is "Yes", will take over described MySQL master data base and continue to provide the MySQL standby database of database service, as described current MySQL database.
18. devices for obtaining MySQL binlog Incremental Log according to claim 17, it is characterized in that, described active-standby switch judging unit specifically for, monitoring be stored in metamessage storehouse, whether change for the field of the IP address of recording MySQL master data base place server; If so, then judge that MySQL database system performs active-standby switch operation.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410125645.8A CN104951474B (en) | 2014-03-31 | 2014-03-31 | Method and device for acquiring MySQL binlog incremental log |
| HK15111889.4A HK1211110A1 (en) | 2014-03-31 | 2015-12-03 | Method for acquiring incremental log of mysql binlog mysqlbinlog |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410125645.8A CN104951474B (en) | 2014-03-31 | 2014-03-31 | Method and device for acquiring MySQL binlog incremental log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104951474A true CN104951474A (en) | 2015-09-30 |
| CN104951474B CN104951474B (en) | 2021-10-01 |
Family
ID=54166137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410125645.8A Active CN104951474B (en) | 2014-03-31 | 2014-03-31 | Method and device for acquiring MySQL binlog incremental log |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104951474B (en) |
| HK (1) | HK1211110A1 (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760456A (en) * | 2016-02-04 | 2016-07-13 | 网易（杭州）网络有限公司 | Method and device for maintaining data consistency |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN105956207A (en) * | 2016-07-01 | 2016-09-21 | 杭州帕拉迪网络科技有限公司 | Binlog-based configurable mysql database real-time synchronization method |
| CN106250496A (en) * | 2016-08-02 | 2016-12-21 | 北京集奥聚合科技有限公司 | A kind of method and system of the data collection in journal file |
| CN106294470A (en) * | 2015-06-03 | 2017-01-04 | 北京京东尚科信息技术有限公司 | The method that real-time incremental log information based on cutting daily record reads |
| CN106294713A (en) * | 2016-08-09 | 2017-01-04 | 深圳中兴网信科技有限公司 | The method of data synchronization resolved based on Incremental Log and data synchronization unit |
| CN106407356A (en) * | 2016-09-07 | 2017-02-15 | 网易（杭州）网络有限公司 | Data backup method and device |
| CN106446239A (en) * | 2016-10-11 | 2017-02-22 | 北京集奥聚合科技有限公司 | Binlog-based data real-time processing method and system |
| CN106911788A (en) * | 2017-03-08 | 2017-06-30 | 青岛海信电器股份有限公司 | A kind of server sends method and device, the terminal of operation note to terminal |
| CN107102934A (en) * | 2016-02-22 | 2017-08-29 | 阿里巴巴集团控股有限公司 | The method and apparatus that a kind of relevant database binary log is reset |
| CN108345684A (en) * | 2018-03-06 | 2018-07-31 | 弘成科技发展有限公司 | Intelligent analysis system and analysis method based on various dimensions multi objective system |
| CN108920522A (en) * | 2018-06-04 | 2018-11-30 | 上海点融信息科技有限责任公司 | Data processing method, system and computer readable storage medium for database |
| CN109388523A (en) * | 2018-09-26 | 2019-02-26 | 四川巧夺天工信息安全智能设备有限公司 | A method of based on binary log file access pattern MySQL database |
| CN109408330A (en) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | Log analysis method, device, terminal device and readable storage medium storing program for executing |
| CN109492012A (en) * | 2018-10-31 | 2019-03-19 | 厦门安胜网络科技有限公司 | A kind of method, apparatus and storage medium of data real-time statistics and retrieval |
| CN110069489A (en) * | 2017-10-17 | 2019-07-30 | 株式会社日立制作所 | A kind of information processing method, device, equipment and computer readable storage medium |
| CN110147355A (en) * | 2017-09-21 | 2019-08-20 | 阿里巴巴集团控股有限公司 | Method of data synchronization, device and server |
| CN110198327A (en) * | 2018-03-05 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of data transmission method and relevant device |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN110750410A (en) * | 2018-07-24 | 2020-02-04 | 北京京东尚科信息技术有限公司 | Method and device for monitoring database logs |
| CN110866022A (en) * | 2019-10-24 | 2020-03-06 | 贝壳技术有限公司 | Data analysis method, system and device based on log file |
| CN110928890A (en) * | 2019-11-08 | 2020-03-27 | 广州华多网络科技有限公司 | Data storage method and device, electronic equipment and computer readable medium |
| CN110990365A (en) * | 2019-12-03 | 2020-04-10 | 北京奇艺世纪科技有限公司 | Data synchronization method, device, server and storage medium |
| CN111078463A (en) * | 2018-10-19 | 2020-04-28 | 阿里巴巴集团控股有限公司 | Data backup method, device and system |
| CN111176887A (en) * | 2018-11-09 | 2020-05-19 | 上海擎感智能科技有限公司 | MySQL misoperation rollback method, equipment and system |
| CN112100139A (en) * | 2020-11-12 | 2020-12-18 | 北京云真信科技有限公司 | Automatic data quality detection system based on big data |
| CN112286729A (en) * | 2020-11-03 | 2021-01-29 | 浪潮云信息技术股份公司 | Method for restoring specified time |
| CN113805806A (en) * | 2018-12-03 | 2021-12-17 | 北京奥星贝斯科技有限公司 | Efficient data unit reusing method and system |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127640A (en) * | 2007-09-10 | 2008-02-20 | 中兴通讯股份有限公司 | A method for improving data synchronization efficiency via the incremental mode |
| CN102915336A (en) * | 2012-09-18 | 2013-02-06 | 北京金和软件股份有限公司 | Incremental data capturing and extraction method based on timestamps and logs |
| CN103297529A (en) * | 2013-06-06 | 2013-09-11 | 浙江大学 | Timestamp-based tree structure data synchronization method |
-
2014
- 2014-03-31 CN CN201410125645.8A patent/CN104951474B/en active Active
-
2015
- 2015-12-03 HK HK15111889.4A patent/HK1211110A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127640A (en) * | 2007-09-10 | 2008-02-20 | 中兴通讯股份有限公司 | A method for improving data synchronization efficiency via the incremental mode |
| CN102915336A (en) * | 2012-09-18 | 2013-02-06 | 北京金和软件股份有限公司 | Incremental data capturing and extraction method based on timestamps and logs |
| CN103297529A (en) * | 2013-06-06 | 2013-09-11 | 浙江大学 | Timestamp-based tree structure data synchronization method |
Non-Patent Citations (2)
| Title |
|---|
| AGAPPLE: ""Canal AdminGuide"", 《HTTP://JU.OUTOFMEMORY.CN/ENTRY/128465》 * |
| ITEYE博客: ""阿里巴巴开源项目：基于mysql数据库binlog的增量订阅&消费"", 《HTTP://AGAPPLE.ITEYE.COM/BLOG/1796633》 * |
Cited By (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294470A (en) * | 2015-06-03 | 2017-01-04 | 北京京东尚科信息技术有限公司 | The method that real-time incremental log information based on cutting daily record reads |
| CN106294470B (en) * | 2015-06-03 | 2020-06-05 | 北京京东尚科信息技术有限公司 | Method for reading real-time incremental log information based on split log |
| CN105760456A (en) * | 2016-02-04 | 2016-07-13 | 网易（杭州）网络有限公司 | Method and device for maintaining data consistency |
| CN105760456B (en) * | 2016-02-04 | 2019-11-29 | 网易（杭州）网络有限公司 | A kind of method and apparatus keeping data consistency |
| CN107102934B (en) * | 2016-02-22 | 2020-12-04 | 阿里巴巴集团控股有限公司 | Method and equipment for replaying binary log of relational database |
| CN107102934A (en) * | 2016-02-22 | 2017-08-29 | 阿里巴巴集团控股有限公司 | The method and apparatus that a kind of relevant database binary log is reset |
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN105912628B (en) * | 2016-04-07 | 2019-05-28 | 北京奇虎科技有限公司 | The synchronous method and device of master-slave database |
| CN105956207A (en) * | 2016-07-01 | 2016-09-21 | 杭州帕拉迪网络科技有限公司 | Binlog-based configurable mysql database real-time synchronization method |
| CN106250496A (en) * | 2016-08-02 | 2016-12-21 | 北京集奥聚合科技有限公司 | A kind of method and system of the data collection in journal file |
| CN106294713A (en) * | 2016-08-09 | 2017-01-04 | 深圳中兴网信科技有限公司 | The method of data synchronization resolved based on Incremental Log and data synchronization unit |
| CN106407356A (en) * | 2016-09-07 | 2017-02-15 | 网易（杭州）网络有限公司 | Data backup method and device |
| CN106446239A (en) * | 2016-10-11 | 2017-02-22 | 北京集奥聚合科技有限公司 | Binlog-based data real-time processing method and system |
| CN106911788A (en) * | 2017-03-08 | 2017-06-30 | 青岛海信电器股份有限公司 | A kind of server sends method and device, the terminal of operation note to terminal |
| CN106911788B (en) * | 2017-03-08 | 2020-06-23 | 海信视像科技股份有限公司 | Method and device for sending operation record to terminal by server and terminal |
| CN110147355A (en) * | 2017-09-21 | 2019-08-20 | 阿里巴巴集团控股有限公司 | Method of data synchronization, device and server |
| CN110069489B (en) * | 2017-10-17 | 2023-01-31 | 株式会社日立制作所 | Information processing method, device and equipment and computer readable storage medium |
| CN110069489A (en) * | 2017-10-17 | 2019-07-30 | 株式会社日立制作所 | A kind of information processing method, device, equipment and computer readable storage medium |
| CN110198327B (en) * | 2018-03-05 | 2021-09-28 | 腾讯科技（深圳）有限公司 | Data transmission method and related equipment |
| CN110198327A (en) * | 2018-03-05 | 2019-09-03 | 腾讯科技（深圳）有限公司 | A kind of data transmission method and relevant device |
| CN108345684A (en) * | 2018-03-06 | 2018-07-31 | 弘成科技发展有限公司 | Intelligent analysis system and analysis method based on various dimensions multi objective system |
| CN108920522A (en) * | 2018-06-04 | 2018-11-30 | 上海点融信息科技有限责任公司 | Data processing method, system and computer readable storage medium for database |
| CN110647421B (en) * | 2018-06-27 | 2022-11-25 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN110647421A (en) * | 2018-06-27 | 2020-01-03 | 阿里巴巴集团控股有限公司 | Database processing method, device and system and electronic equipment |
| CN110750410A (en) * | 2018-07-24 | 2020-02-04 | 北京京东尚科信息技术有限公司 | Method and device for monitoring database logs |
| CN110750410B (en) * | 2018-07-24 | 2024-04-12 | 北京京东尚科信息技术有限公司 | Method and device for monitoring database logs |
| CN109388523B (en) * | 2018-09-26 | 2021-11-30 | 四川巧夺天工信息安全智能设备有限公司 | Method for recovering MySQL database based on binary log file |
| CN109388523A (en) * | 2018-09-26 | 2019-02-26 | 四川巧夺天工信息安全智能设备有限公司 | A method of based on binary log file access pattern MySQL database |
| CN109408330A (en) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | Log analysis method, device, terminal device and readable storage medium storing program for executing |
| CN111078463B (en) * | 2018-10-19 | 2023-05-02 | 阿里云计算有限公司 | Data backup method, device and system |
| CN111078463A (en) * | 2018-10-19 | 2020-04-28 | 阿里巴巴集团控股有限公司 | Data backup method, device and system |
| CN109492012A (en) * | 2018-10-31 | 2019-03-19 | 厦门安胜网络科技有限公司 | A kind of method, apparatus and storage medium of data real-time statistics and retrieval |
| CN111176887A (en) * | 2018-11-09 | 2020-05-19 | 上海擎感智能科技有限公司 | MySQL misoperation rollback method, equipment and system |
| CN113805806A (en) * | 2018-12-03 | 2021-12-17 | 北京奥星贝斯科技有限公司 | Efficient data unit reusing method and system |
| CN110866022A (en) * | 2019-10-24 | 2020-03-06 | 贝壳技术有限公司 | Data analysis method, system and device based on log file |
| CN110928890A (en) * | 2019-11-08 | 2020-03-27 | 广州华多网络科技有限公司 | Data storage method and device, electronic equipment and computer readable medium |
| CN110928890B (en) * | 2019-11-08 | 2023-01-24 | 广州华多网络科技有限公司 | Data storage method and device, electronic equipment and computer readable storage medium |
| CN110990365A (en) * | 2019-12-03 | 2020-04-10 | 北京奇艺世纪科技有限公司 | Data synchronization method, device, server and storage medium |
| CN112286729A (en) * | 2020-11-03 | 2021-01-29 | 浪潮云信息技术股份公司 | Method for restoring specified time |
| CN112286729B (en) * | 2020-11-03 | 2023-02-21 | 浪潮云信息技术股份公司 | Method for restoring specified time |
| CN112100139B (en) * | 2020-11-12 | 2021-02-09 | 北京云真信科技有限公司 | Automatic data quality detection system based on big data |
| CN112100139A (en) * | 2020-11-12 | 2020-12-18 | 北京云真信科技有限公司 | Automatic data quality detection system based on big data |
| CN114489995B (en) * | 2022-02-15 | 2022-09-30 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
| CN114489995A (en) * | 2022-02-15 | 2022-05-13 | 北京永信至诚科技股份有限公司 | Distributed scheduling processing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| HK1211110A1 (en) | 2016-05-13 |
| CN104951474B (en) | 2021-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104951474A (en) | Method and device for acquiring MySQL binlog incremental logs | |
| US20160364407A1 (en) | Method and Device for Responding to Request, and Distributed File System | |
| CN108932295A (en) | Primary database method for handover control, device, computer equipment and storage medium | |
| CN109600410B (en) | Data storage system and method | |
| US20070143286A1 (en) | File management method in file system and metadata server therefor | |
| CN111400312B (en) | Edge storage database based on improved LSM tree | |
| CN109063192B (en) | Working method of high-performance mass file storage system | |
| CN104020961A (en) | Distributed data storage method, device and system | |
| CN111355791B (en) | File transmission method and device, computer equipment and storage medium | |
| CN111524005B (en) | Handle identification system based on blockchain and data processing method | |
| US20130325932A1 (en) | Electronic device and method for storing distributed documents | |
| CN109522316A (en) | Log processing method, device, equipment and storage medium | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| CN110008197A (en) | A kind of data processing method, system and electronic equipment and storage medium | |
| CN113806301A (en) | Data synchronization method, device, server and storage medium | |
| CN112748877A (en) | File integration uploading method and device and file downloading method and device | |
| CN103927252A (en) | Cross-component log recording method, device and system | |
| CN102073733A (en) | Method and device for managing Hash table | |
| CN113448946B (en) | Data migration method and device and electronic equipment | |
| CN108900337B (en) | Fault recovery method, server, client and system for metadata service | |
| WO2021254214A1 (en) | Migration method and apparatus for android application program | |
| CN109445988B (en) | Heterogeneous disaster recovery method, device, system, server and disaster recovery platform | |
| CN114840488B (en) | Distributed storage method, system and storage medium based on super fusion structure | |
| US9002788B2 (en) | System for configurable reporting of network data and related method | |
| CN113590380A (en) | Database recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1211110 Country of ref document: HK |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |