CN109857802A - Daily record data synchronous method, device, equipment and computer readable storage medium - Google Patents
Daily record data synchronous method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN109857802A CN109857802A CN201811522747.8A CN201811522747A CN109857802A CN 109857802 A CN109857802 A CN 109857802A CN 201811522747 A CN201811522747 A CN 201811522747A CN 109857802 A CN109857802 A CN 109857802A
- Authority
- CN
- China
- Prior art keywords
- daily record
- record data
- gtid
- aid
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 39
- 238000012544 monitoring process Methods 0.000 claims abstract description 30
- 230000002159 abnormal effect Effects 0.000 claims abstract description 29
- 230000000737 periodic effect Effects 0.000 claims description 12
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of daily record data synchronous method, comprising: in the synchronizing process of daily record data, often collects a daily record data, then obtains the GTID of the daily record data, and generate the AID of the daily record data；The daily record data, the GTID and the AID are synchronously written to cache database Redis and distributed post and subscribe to message system Kafka；When the current acquisition node for monitoring daily record data occurs abnormal, the target acquisition node of daily record data is determined, and the target GTID and target AID of the acquisition time daily record data nearest apart from current point in time are obtained from the Redis；By the target acquisition node, continue to acquire daily record data based on the target GTID and the target AID.The invention also discloses a kind of daily record data synchronizing device, equipment and computer readable storage mediums.The present invention can guarantee the continuity of daily record data.
Description
Technical field
The present invention relates to data acquisition technical field more particularly to a kind of daily record data synchronous method, device, equipment and
Computer readable storage medium.
Background technique
Database in the process of running, needs the daily record data of acquisition database, and the daily record data of database is adopted at present
Mode set includes the log data acquisition based on Maxwell tool and the log data acquisition based on Canal tool, above two
Mode is the acquisition that daily record data is realized from library by the MySQL Server that disguises oneself as.However, in the acquisition of daily record data
In the process, when the tool of acquisition occurs abnormal, being easy to appear collected daily record data, there are discontinuous problems, therefore,
How to guarantee that the continuity of daily record data is current urgent problem to be solved.
Summary of the invention
The main purpose of the present invention is to provide a kind of daily record data synchronous method, device, equipment and computer-readable deposit
Storage media, it is intended to guarantee the continuity of daily record data.
To achieve the above object, the present invention provides a kind of daily record data synchronous method, the daily record data synchronous method packet
Include following steps:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the overall situation of the daily record data
Transaction identifiers GTID, and generate the atom identifier AID of the daily record data；
The daily record data, the GTID and the AID are synchronously written to cache database Redis and distributed post
Subscribe to message system Kafka；
When the current acquisition node for monitoring daily record data occurs abnormal, the target acquisition node of daily record data is determined,
And the target GTID and target AID of the acquisition time daily record data nearest apart from current point in time are obtained from the Redis；
By the target acquisition node, continue to acquire daily record data based on the target GTID and the target AID.
Further, the step of generating the atom identifier AID of the daily record data include:
Often collect a daily record data, then obtain the AID of a daily record data, and on the basis of in the AID from
Increase 1, obtains the AID of the daily record data.
Further, described when the current acquisition node for monitoring daily record data occurs abnormal, determine daily record data
The step of target acquisition node includes:
When the current acquisition node for monitoring daily record data occurs abnormal, each acquisition node of daily record data is obtained, and
Obtain the registration order that each acquisition node respectively corresponds to the child node of registration；
The registration order that the child node of registration is respectively corresponded to according to each acquisition node determines the target acquisition section of daily record data
Point.
Further, the daily record data synchronous method further include:
In the synchronizing process of daily record data, whether the database that periodic monitor currently connects is primary database；
If monitoring, the database currently connected is primary database, disconnects the connection with the primary database, and obtain
Standby database address, and corresponding standby database is connected according to the standby database address.
Further, the daily record data synchronous method further include:
In the synchronizing process of daily record data, whether the standby database that periodic monitor currently connects there is exception；
If there is abnormal, the new standby database address of acquisition in the standby database currently connected, and according to new standby data
Library address connects corresponding standby database.
Further, the daily record data synchronous method further include:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the first of the daily record data
GTID, and obtain from the Redis the 2nd GTID of upper one article of daily record data；
Judge whether the first GTID and the 2nd GTID are adjacent；
If the first GTID and the 2nd GTID are adjacent, the AID of the daily record data is generated, and by the log
Data, the first GTID and AID are synchronously written to the Redis and the Kafka；
If the first GTID and the 2nd GTID are non-conterminous, continue to acquire log number according to the 2nd GTID
According to.
In addition, to achieve the above object, the present invention also provides a kind of daily record data synchronizing device, the daily record data is synchronous
Device includes:
Data acquisition module then obtains institute in the synchronizing process of daily record data, often collecting a daily record data
The global transaction identifier GTID of daily record data is stated, and generates the atom identifier AID of the daily record data；
Data simultaneous module, for being synchronously written the daily record data, the GTID and the AID to cache database
Redis and distributed post subscribe to message system Kafka；
Determining module, for determining daily record data when the current acquisition node for monitoring daily record data occurs abnormal
Target acquisition node, and obtain from the Redis target of the acquisition time daily record data nearest apart from current point in time
GTID and target AID；
The data acquisition module is also used to be based on the target GTID and the mesh by the target acquisition node
Mark AID continues to acquire daily record data.
In addition, to achieve the above object, the present invention also provides a kind of daily record data synchronizer, the daily record data is synchronous
Equipment includes: memory, processor and to be stored in the daily record data that can be run on the memory and on the processor same
Program is walked, the daily record data synchronization program realizes daily record data synchronous method as described above when being executed by the processor
Step.
The present invention also provides a kind of computer readable storage medium, log is stored on the computer readable storage medium
Data synchronization program, the daily record data synchronization program realize daily record data synchronous method as described above when being executed by processor
The step of.
The present invention provides a kind of daily record data synchronous method, device, equipment and computer readable storage medium, the present invention and exists
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the GTID of the daily record data, and generates the log
Then the daily record data, GTID and AID are synchronously written to Redis and Kafka by the AID of data, then when monitoring log number
According to current acquisition node occur abnormal when, determine the target acquisition node of daily record data, and acquisition time is obtained from Redis
The target GTID and target AID of the daily record data nearest apart from current point in time, finally by the target acquisition node, being based on should
Target GTID and target AID continue acquire daily record data, due to by daily record data, GTID and AID be synchronously written Redis and
Kafka can directly obtain corresponding target GTID when the current acquisition node of daily record data occurs abnormal from Redis
Continue to acquire daily record data based on target GTID and target AID, prevent acquisition from saving then by target acquisition node with target AID
Caused data acquisition is interrupted when point exception, and the present invention can continuously acquire daily record data, effectively guarantee daily record data
Continuity.
Detailed description of the invention
Fig. 1 is the device structure schematic diagram for the hardware running environment that the embodiment of the present invention is related to；
Fig. 2 is the flow diagram of daily record data synchronous method first embodiment of the present invention；
Fig. 3 is the functional block diagram of daily record data synchronizing device first embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
As shown in Figure 1, Fig. 1 is the device structure schematic diagram for the hardware running environment that the embodiment of the present invention is related to.
Daily record data synchronizer of the embodiment of the present invention can be PC, be also possible to smart phone, tablet computer, portable meter
The packaged type terminal device having a display function such as calculation machine.
As shown in Figure 1, the daily record data synchronizer may include: processor 1001, such as CPU, communication bus 1002,
User interface 1003, network interface 1004, memory 1005.Wherein, communication bus 1002 is for realizing between these components
Connection communication.User interface 1003 may include display screen (Display), input unit such as keyboard (Keyboard), optional
User interface 1003 can also include standard wireline interface and wireless interface.Network interface 1004 optionally may include mark
Wireline interface, the wireless interface (such as WI-FI interface) of standard.Memory 1005 can be high speed RAM memory, be also possible to stablize
Memory (non-volatile memory), such as magnetic disk storage.Memory 1005 optionally can also be independently of preceding
State the storage device of processor 1001.
It will be understood by those skilled in the art that daily record data synchronizer structure shown in Fig. 1 is not constituted to log
The restriction of data synchronizer may include perhaps combining certain components or different than illustrating more or fewer components
Component layout.
As shown in Figure 1, as may include that operating system, network are logical in a kind of memory 1005 of computer storage medium
Believe module, Subscriber Interface Module SIM and daily record data synchronization program.
In daily record data synchronizer shown in Fig. 1, network interface 1004 is mainly used for connecting background server, and rear
Platform server carries out data communication；User interface 1003 is mainly used for connecting client (user terminal), carries out data with client
Communication；And processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, and execute following step
It is rapid:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the overall situation of the daily record data
Transaction identifiers GTID, and generate the atom identifier AID of the daily record data；
The daily record data, the GTID and the AID are synchronously written to cache database Redis and distributed post
Subscribe to message system Kafka；
When the current acquisition node for monitoring daily record data occurs abnormal, the target acquisition node of daily record data is determined,
And the target GTID and target AID of the acquisition time daily record data nearest apart from current point in time are obtained from the Redis；
By the target acquisition node, continue to acquire daily record data based on the target GTID and the target AID.
Further, processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, also
Execute following steps:
Often collect a daily record data, then obtain the AID of a daily record data, and on the basis of in the AID from
Increase 1, obtains the AID of the daily record data.
Further, processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, also
Execute following steps:
When the current acquisition node for monitoring daily record data occurs abnormal, each acquisition node of daily record data is obtained, and
Obtain the registration order that each acquisition node respectively corresponds to the child node of registration；
The registration order that the child node of registration is respectively corresponded to according to each acquisition node determines the target acquisition section of daily record data
Point.
Further, processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, also
Execute following steps:
In the synchronizing process of daily record data, whether the database that periodic monitor currently connects is primary database；
If monitoring, the database currently connected is primary database, disconnects the connection with the primary database, and obtain
Standby database address, and corresponding standby database is connected according to the standby database address.
Further, processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, also
Execute following steps:
In the synchronizing process of daily record data, whether the standby database that periodic monitor currently connects there is exception；
If there is abnormal, the new standby database address of acquisition in the standby database currently connected, and according to new standby data
Library address connects corresponding standby database.
Further, processor 1001 can be used for calling the daily record data synchronization program stored in memory 1005, also
Execute following steps:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the first of the daily record data
GTID, and obtain from the Redis the 2nd GTID of upper one article of daily record data；
Judge whether the first GTID and the 2nd GTID are adjacent；
If the first GTID and the 2nd GTID are adjacent, the AID of the daily record data is generated, and by the log
Data, the first GTID and AID are synchronously written to the Redis and the Kafka；
If the first GTID and the 2nd GTID are non-conterminous, continue to acquire log number according to the 2nd GTID
According to.
Wherein, the specific embodiment of daily record data synchronizer of the present invention and following daily record data synchronous method is each specific
Embodiment is essentially identical, and therefore not to repeat here.
The present invention provides a kind of daily record data synchronous method.
It is the flow diagram of daily record data synchronous method first embodiment of the present invention referring to Fig. 2, Fig. 2.
In the present embodiment, which includes:
Step S101 often collects a daily record data, then obtains daily record data in the synchronizing process of daily record data
Global transaction identifier GTID, and generate the atom identifier AID of daily record data；
Step S102, the daily record data, the GTID and the AID are synchronously written to cache database Redis and
Distributed post subscribes to message system Kafka；
In the present embodiment, the daily record data synchronous method is optional to be applied to daily record data synchronizer, the daily record data
Synchronizer daily record data synchronizer as described above.The daily record data synchronizer can integrate daily record data to be collected
Active/standby database, the cluster for being deployed with log data acquisition program, distributed application program coordination service ZooKeeper, caching
Database Redis and distributed post subscribe to message system Kafka.
Further, in order to prevent individual equipment failure when the synchronization of caused daily record data interrupt, it is of the invention in, institute
It states daily record data synchronous method and applies also for daily record data synchronization system, which includes log to be collected
The active/standby database of data, the cluster for being deployed with log data acquisition program, distributed application program coordination service
ZooKeeper, cache database Redis and distributed post subscribe to message system Kafka.Wherein daily record data to be collected
Active/standby database includes but is not limited to TDSQL, MySQL and MariaDB, two servers is included at least in the cluster, and each
Log data acquisition program is deployed in server, which is used to guarantee the High Availabitity of log data acquisition program, no
There is single-point problem, the Redis and Kafka for storing collected daily record data.The ZooKeeper is for guaranteeing log number
According to the High Availabitity of capture program, do not occur single-point problem specific manifestation are as follows: each server starting log data in the cluster
When capture program, the server of each starting log capture program registers an interim orderly child node to ZooKeeper,
Then the corresponding child node of each log collection program is obtained from ZooKeeper, and each child node is carried out according to registion time
Sequence, the corresponding server of the smallest child node that will sort is determined as log data acquisition node, or the maximum son that will sort
The corresponding server of node is determined as log data acquisition node, is arranged with specific reference to actual needs, hereafter preferably to sort most
The corresponding server of small child node is determined as log data acquisition node.
After determining the current acquisition node of daily record data, by current acquisition node from primary database/standby database
Middle acquisition daily record data, and collected daily record data is synchronously written Kafka and Redis, specially in the same of daily record data
During step, log data acquisition node often collects a daily record data, then obtains the GTID (Global of the daily record data
Transaction Identifier, global transaction identifier), and generate the AID (Atomic of the daily record data
Identifier, atom identifier), then by collected daily record data, the GTID of the daily record data and AID be synchronously written to
Cache database Redis and distributed post subscribe to message system Kafka.Wherein, the generating mode of the AID are as follows: often collect
One daily record data then obtains the AID of a daily record data, and increases 1 certainly on the basis of the AID of a upper daily record data,
The AID of current collected daily record data is obtained, the initial value of the AID is preset constant (such as 0 or 1), wherein AID is current
Acquisition node for the first time starting using preceding addition field, the field for current acquisition node starting after to daily record data whether
Missing is judged that the AID between each continuous daily record data of item is continuous.For example, the AID of a upper daily record data is
1236, then the AID of the current collected daily record data generated is 1237.
Step S103 determines the target of daily record data when the current acquisition node for monitoring daily record data occurs abnormal
Acquisition node, and obtain from Redis the target GTID and target of the acquisition time daily record data nearest apart from current point in time
AID；
In the present embodiment, in the synchronizing process of daily record data, pass through the current acquisition of ZooKeeper monitoring journal data
Whether node there is exception, when the current acquisition node for monitoring daily record data occurs abnormal, determines the target of daily record data
Acquisition node, and obtain from the Redis target GTID and mesh of the acquisition time daily record data nearest apart from current point in time
Mark AID.Wherein, whether ZooKeeper there is exception by the current acquisition node of watch interface monitor daily record data, specifically
For by whether being stored with the corresponding registration child node of current acquisition node in watch interface monitoring ZooKeeper, if
It is stored with the corresponding registration child node of current acquisition node in ZooKeeper, then is determining the current acquisition node of daily record data just
Often, if not stored in ZooKeeper have the corresponding registration child node of current acquisition node, determine currently adopting for daily record data
Collect node and exception occurs.It should be noted that when the current acquisition node of daily record data occurs abnormal, in the ZooKeeper
The correspondence child node of registration can be deleted, i.e. the corresponding registration of current acquisition node of daily record data is not present in ZooKeeper
Child node.
Wherein, the method for determination of target acquisition node is specially when the current acquisition node appearance for monitoring daily record data is different
Chang Shi, obtains each acquisition node of daily record data, and obtains each acquisition node respectively corresponding registration order for registering child node, so
The registration order for respectively corresponding to the child node of registration according to each acquisition node afterwards, determines the target acquisition node of daily record data, i.e.,
The registration order that the child node of registration is respectively corresponded to according to each acquisition node, is ranked up the child node of each registration, and will row
The corresponding acquisition node of the smallest child node of sequence is determined as the target acquisition node of daily record data.That is, from day is deployed with
Select a server as target acquisition node in the cluster of will data acquisition program.
Step S104 continues to acquire daily record data based on target GTID and target AID by target acquisition node.
In the present embodiment, after determining target acquisition node, target GTID and target AID, by target acquisition node,
Continue to acquire daily record data based on target GTID and target AID, i.e., acquires log number from active/standby database according to target GTID
According to, and the GTID of collected daily record data is obtained, it is then based on the AID that target AID generates the daily record data from increasing 1,
And the GTID and AID of collected daily record data and the daily record data are synchronously written to Redis and Kafka.For example, mesh
Marking AID is 5423, then the AID of the daily record data generated based on target is 5424.Wherein, by daily record data, daily record data
GTID and AID is synchronously written to Redis and Kafka, Redis and Kafka can timing duplicate removal is carried out to daily record data
Processing specially obtains the AID of every daily record data, and the duplicate daily record data of AID is deleted, and avoids the weight of daily record data
It is multiple.
In specific implementation, operation system obtains the daily record data read out when reading daily record data from Kafka
First AID, and the 2nd AID of the daily record data that one article reads out is obtained, then judge between the first AID and the 2nd AID
Whether have a hopping sequence, i.e. difference between calculating the first AID and the 2nd AID, if the difference is 1, can determine the first AID and
There is no hopping sequence between 2nd AID, if the difference is more than or equal to 2, can determine hopping sequence between the first AID and the 2nd AID；Such as
Hopping sequence between fruit the first AID and the 2nd AID, then calculate the AID skipped, and the AID by skipping obtained from Redis it is corresponding
Daily record data, and the daily record data supplement that will acquire is written to Kafka.Can there can be the case where missing in daily record data
Under, the daily record data of missing is quickly obtained, guarantees the continuity of daily record data.It should be noted that existing mode, generally
It is that data deduplication is carried out by business major key or unique index, it is not general enough when in use, usually the shortage of data the case where
Under, it is more difficult to judge whether there is missing.Even if needing to judge missing, it is also desirable to be judged by continuous major key, if major key
Missing, then be unable to judge accurately whether daily record data lacks.
In the present embodiment, the present invention often collects a daily record data in the synchronizing process of daily record data, then obtaining should
The GTID of daily record data, and generate the AID of the daily record data, then by the daily record data, GTID and AID be synchronously written to
Redis and Kafka determines the target of daily record data then when the current acquisition node for monitoring daily record data occurs abnormal
Acquisition node, and obtain from Redis the target GTID and target of the acquisition time daily record data nearest apart from current point in time
AID continues to acquire daily record data based on target GTID and target AID finally by the target acquisition node, due to by log
Data, GTID and AID are synchronously written Redis and Kafka, can be when the current acquisition node of daily record data occurs abnormal, directly
It connects and obtains corresponding target GTID and target AID from Redis, then by target acquisition node, be based on target GTID and target
AID continues to acquire daily record data, prevents caused data acquisition when acquisition node exception from interrupting, the present invention can be acquired continuously
Daily record data, the effective continuity for guaranteeing daily record data.
Further, implement based on above-mentioned first, propose the second embodiment of daily record data synchronous method of the present invention, with
The difference of previous embodiment is, in the synchronizing process of daily record data, database that periodic monitor currently connects whether based on
Database disconnects the connection with primary database if monitoring that the database currently connected is primary database, and obtains standby number
Corresponding standby database is connected according to the address in library, and according to the address for database.Wherein, primary database is used for processing business
The service request of system, for the total data for being stored with primary database in database, for backing up primary database, by standby number
Log data acquisition is carried out according to library, it can influence to avoid log data acquisition to primary database.
Further, in the synchronizing process of daily record data, go back whether the standby database that periodic monitor currently connects occurs
It is abnormal, if there is abnormal, the new standby database address of acquisition in the standby database currently connected, and according to new standby database
Address connects corresponding standby database.In specific implementation, when the address for monitoring standby database is changed, after obtaining change
Standby database address, and connect standby database according to the standby database address after change.Wherein, the time of above-mentioned timing can be by
Those skilled in the art are configured based on actual conditions, and the present embodiment is not especially limited this.
In the present embodiment, the present invention, can be to avoid log data acquisition by carrying out log data acquisition to standby database
Influence to primary database can also guarantee that collected daily record data is consistent with the daily record data in primary database.
Further, above-mentioned first or second embodiments are based on, the third of daily record data synchronous method of the present invention is proposed
Embodiment, the difference with previous embodiment are, in the synchronizing process of daily record data, often collect a daily record data, then
The first GTID of the daily record data is obtained, and obtains the 2nd GTID of upper one article of daily record data from the Redis, then judges
Whether one GTID and the 2nd GTID are adjacent, i.e. difference between calculating the first GTID and the 2nd GTID, and whether judge the difference
Determine that the first GTID and the 2nd GTID is adjacent, if the difference is not pre- if the difference is preset difference value for preset difference value
If difference, then determine that the first GTID and the 2nd GTID is non-conterminous；If the first GTID and the 2nd GTID are adjacent, to a upper day
The AID of will data is from increasing 1 to generate the AID of the daily record data, and by the daily record data, the first GTID and the AID synchronous write
Enter to Redis and Kafka；If the first GTID and the 2nd GTID are non-conterminous, continue to acquire daily record data according to the 2nd GTID,
Daily record data is acquired from standby database according to the 2nd GTID, and obtains the GTID of collected daily record data, is then obtained
The AID of a upper daily record data, and collected daily record data is obtained from increasing 1 on the basis of the AID of a upper daily record data
Then collected daily record data and corresponding GTID and AID are synchronously written to Redis and Kafka by AID.
In the present embodiment, the present invention often collects a daily record data, then judges current collected daily record data
Whether the GTID of GTID and a upper daily record data is adjacent, if adjacent, daily record data and its corresponding GTID and AID are write
Enter Redis and Kafka, if non-conterminous, the GTID according to a upper daily record data acquires daily record data, can effectively protect
Demonstrate,prove the continuity of daily record data.
The present invention also provides a kind of daily record data synchronizing devices.
It is the functional block diagram of daily record data synchronizing device first embodiment of the present invention referring to Fig. 3, Fig. 3.
In the present embodiment, which includes:
Data acquisition module 101 is then obtained in the synchronizing process of daily record data, often collecting a daily record data
The global transaction identifier GTID of the daily record data is taken, and generates the atom identifier AID of the daily record data；
Data simultaneous module 102, for being synchronously written the daily record data, the GTID and the AID to caching number
Message system Kafka is subscribed to according to library Redis and distributed post；
Determining module 103, for determining daily record data when the current acquisition node for monitoring daily record data occurs abnormal
Target acquisition node, and obtain from the Redis target of the acquisition time daily record data nearest apart from current point in time
GTID and target AID；
The data acquisition module 101, is also used to through the target acquisition node, based on the target GTID and described
Target AID continues to acquire daily record data.
Further, the data acquisition module 101 is also used to:
Often collect a daily record data, then obtain the AID of a daily record data, and on the basis of in the AID from
Increase 1, obtains the AID of the daily record data.
Further, the determining module 103 is also used to:
When the current acquisition node for monitoring daily record data occurs abnormal, each acquisition node of daily record data is obtained, and
Obtain the registration order that each acquisition node respectively corresponds to the child node of registration；
The registration order that the child node of registration is respectively corresponded to according to each acquisition node determines the target acquisition section of daily record data
Point.
Further, the daily record data synchronizing device further include:
Monitoring modular, for database that in the synchronizing process of daily record data, periodic monitor is currently connected whether based on
Database；
Link block disconnects and the primary database if the database for monitoring currently to connect is primary database
Connection, and obtain standby database address, and according to the corresponding standby database of the standby database address connection.
Further, the monitoring modular is also used in the synchronizing process of daily record data, what periodic monitor currently connected
Whether standby database there is exception；
The link block, if exception occurs in the standby database for being also used to currently connect, with obtaining new standby database
Location, and corresponding standby database is connected according to new standby database address.
Further, the daily record data synchronizing device further include:
The data acquisition module, is also used in the synchronizing process of daily record data, often collects a daily record data, then
The first GTID of the daily record data is obtained, and obtains the 2nd GTID of upper one article of daily record data from the Redis；
Judgment module, for judging whether the first GTID and the 2nd GTID are adjacent；
The data simultaneous module generates the log if adjacent for the first GTID and the 2nd GTID
The AID of data, and the daily record data, the first GTID and AID are synchronously written to the Redis and described
Kafka；
The data acquisition module, if being also used to the first GTID and the 2nd GTID is non-conterminous, according to described in
2nd GTID continues to acquire daily record data.
Wherein, the specific embodiment of daily record data synchronizing device of the present invention and above-mentioned each embodiment of daily record data synchronous method
Essentially identical, therefore not to repeat here.
In addition, the embodiment of the present invention also proposes a kind of computer readable storage medium, the computer readable storage medium
On be stored with daily record data synchronization program, when the daily record data synchronization program is executed by processor, execute following steps:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the overall situation of the daily record data
Transaction identifiers GTID, and generate the atom identifier AID of the daily record data；
The daily record data, the GTID and the AID are synchronously written to cache database Redis and distributed post
Subscribe to message system Kafka；
When the current acquisition node for monitoring daily record data occurs abnormal, the target acquisition node of daily record data is determined,
And the target GTID and target AID of the acquisition time daily record data nearest apart from current point in time are obtained from the Redis；
By the target acquisition node, continue to acquire daily record data based on the target GTID and the target AID.
Further, when the daily record data synchronization program is executed by processor, following steps are also executed:
Often collect a daily record data, then obtain the AID of a daily record data, and on the basis of in the AID from
Increase 1, obtains the AID of the daily record data.
Further, when the daily record data synchronization program is executed by processor, following steps are also executed:
When the current acquisition node for monitoring daily record data occurs abnormal, each acquisition node of daily record data is obtained, and
Obtain the registration order that each acquisition node respectively corresponds to the child node of registration；
The registration order that the child node of registration is respectively corresponded to according to each acquisition node determines the target acquisition section of daily record data
Point.
Further, when the daily record data synchronization program is executed by processor, following steps are also executed:
In the synchronizing process of daily record data, whether the database that periodic monitor currently connects is primary database；
If monitoring, the database currently connected is primary database, disconnects the connection with the primary database, and obtain
Standby database address, and corresponding standby database is connected according to the standby database address.
Further, when the daily record data synchronization program is executed by processor, following steps are also executed:
In the synchronizing process of daily record data, whether the standby database that periodic monitor currently connects there is exception；
If there is abnormal, the new standby database address of acquisition in the standby database currently connected, and according to new standby data
Library address connects corresponding standby database.
Further, when the daily record data synchronization program is executed by processor, following steps are also executed:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the first of the daily record data
GTID, and obtain from the Redis the 2nd GTID of upper one article of daily record data；
Judge whether the first GTID and the 2nd GTID are adjacent；
If the first GTID and the 2nd GTID are adjacent, the AID of the daily record data is generated, and by the log
Data, the first GTID and AID are synchronously written to the Redis and the Kafka；
If the first GTID and the 2nd GTID are non-conterminous, continue to acquire log number according to the 2nd GTID
According to.
Wherein, the specific embodiment of computer readable storage medium of the present invention is respectively implemented with above-mentioned daily record data synchronous method
Example is essentially identical, and therefore not to repeat here.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in one as described above
In storage medium (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that terminal device (it can be mobile phone,
Computer, server, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of daily record data synchronous method, which is characterized in that the daily record data synchronous method the following steps are included:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the global transaction of the daily record data
Identifier GTID, and generate the atom identifier AID of the daily record data；
The daily record data, the GTID and the AID are synchronously written to cache database Redis and distributed post subscription
Message system Kafka；
When the current acquisition node for monitoring daily record data occurs abnormal, the target acquisition node of daily record data is determined, and from
The target GTID and target AID of the acquisition time daily record data nearest apart from current point in time are obtained in the Redis；
By the target acquisition node, continue to acquire daily record data based on the target GTID and the target AID.
2. daily record data synchronous method as described in claim 1, which is characterized in that generate the atom mark of the daily record data
Accord with AID the step of include:
A daily record data is often collected, then obtains the AID of a daily record data, and from increasing 1 on the basis of in the AID,
Obtain the AID of the daily record data.
3. daily record data synchronous method as described in claim 1, which is characterized in that described to monitor the current of daily record data
When acquisition node occurs abnormal, the step of determining the target acquisition node of daily record data, includes:
When the current acquisition node for monitoring daily record data occurs abnormal, each acquisition node of daily record data is obtained, and obtain
Each acquisition node respectively corresponds to the registration order of the child node of registration；
The registration order that the child node of registration is respectively corresponded to according to each acquisition node, determines the target acquisition node of daily record data.
4. daily record data synchronous method as described in claim 1, which is characterized in that the daily record data synchronous method also wraps
It includes:
In the synchronizing process of daily record data, whether the database that periodic monitor currently connects is primary database；
If monitoring, the database currently connected is primary database, disconnects the connection with the primary database, and obtains standby number
Corresponding standby database is connected according to library address, and according to the standby database address.
5. daily record data synchronous method as claimed in claim 4, which is characterized in that the daily record data synchronous method also wraps
It includes:
In the synchronizing process of daily record data, whether the standby database that periodic monitor currently connects there is exception；
If there is abnormal, the new standby database address of acquisition in the standby database currently connected, and according to new standby database
Location connects corresponding standby database.
6. daily record data synchronous method according to any one of claims 1 to 5, which is characterized in that the daily record data is synchronous
Method further include:
In the synchronizing process of daily record data, a daily record data is often collected, then obtains the first GTID of the daily record data,
And the 2nd GTID of upper one article of daily record data is obtained from the Redis；
Judge whether the first GTID and the 2nd GTID are adjacent；
If the first GTID and the 2nd GTID are adjacent, the AID of the daily record data is generated, and by the log number
It is synchronously written according to, the first GTID and AID to the Redis and the Kafka；
If the first GTID and the 2nd GTID are non-conterminous, continue to acquire daily record data according to the 2nd GTID.
7. a kind of daily record data synchronizing device, which is characterized in that the daily record data synchronizing device includes:
Data acquisition module then obtains the day in the synchronizing process of daily record data, often collecting a daily record data
The global transaction identifier GTID of will data, and generate the atom identifier AID of the daily record data；
Data simultaneous module, for being synchronously written the daily record data, the GTID and the AID to cache database
Redis and distributed post subscribe to message system Kafka；
Determining module, for determining the target of daily record data when the current acquisition node for monitoring daily record data occurs abnormal
Acquisition node, and obtain from the Redis acquisition time daily record data nearest apart from current point in time target GTID and
Target AID；
The data acquisition module is also used to be based on the target GTID and target AID by the target acquisition node
Continue to acquire daily record data.
8. daily record data synchronizing device as claimed in claim 7, which is characterized in that the daily record data synchronizing device also wraps
It includes:
The data acquisition module, is also used in the synchronizing process of daily record data, often collects a daily record data, then obtains
First GTID of the daily record data, and obtain from the Redis the 2nd GTID of upper one article of daily record data；
Judgment module, for judging whether the first GTID and the 2nd GTID are adjacent；
The data simultaneous module generates the daily record data if adjacent for the first GTID and the 2nd GTID
AID, and the daily record data, the first GTID and AID are synchronously written to the Redis and the Kafka；
The data acquisition module, if being also used to the first GTID and the 2nd GTID is non-conterminous, according to described second
GTID continues to acquire daily record data.
9. a kind of daily record data synchronizer, which is characterized in that the daily record data synchronizer includes: memory, processor
And it is stored in the daily record data synchronization program that can be run on the memory and on the processor, the daily record data is synchronous
It realizes when program is executed by the processor such as the step of daily record data synchronous method described in any one of claims 1 to 6.
10. a kind of computer readable storage medium, which is characterized in that be stored with log number on the computer readable storage medium
According to synchronization program, realize when the daily record data synchronization program is executed by processor as described in any one of claims 1 to 6
The step of daily record data synchronous method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811522747.8A CN109857802A (en) | 2018-12-12 | 2018-12-12 | Daily record data synchronous method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811522747.8A CN109857802A (en) | 2018-12-12 | 2018-12-12 | Daily record data synchronous method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109857802A true CN109857802A (en) | 2019-06-07 |
Family
ID=66890980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811522747.8A Pending CN109857802A (en) | 2018-12-12 | 2018-12-12 | Daily record data synchronous method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109857802A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110471774A (en) * | 2019-06-28 | 2019-11-19 | 苏宁云计算有限公司 | A kind of data processing method and device based on unified task schedule |
| CN112395157A (en) * | 2020-11-13 | 2021-02-23 | 广州至真信息科技有限公司 | Audit log obtaining method and device, computer equipment and storage medium |
| CN114666753A (en) * | 2020-12-04 | 2022-06-24 | 中国移动通信集团广东有限公司 | Signaling stream data processing method, position information service method and electronic equipment |
| WO2022134876A1 (en) * | 2020-12-24 | 2022-06-30 | 中兴通讯股份有限公司 | Data synchronization method and apparatus, and electronic device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040030703A1 (en) * | 2002-08-12 | 2004-02-12 | International Business Machines Corporation | Method, system, and program for merging log entries from multiple recovery log files |
| CN106407356A (en) * | 2016-09-07 | 2017-02-15 | 网易（杭州）网络有限公司 | Data backup method and device |
| CN107291787A (en) * | 2016-04-13 | 2017-10-24 | 中兴通讯股份有限公司 | Master/slave data storehouse switching method and apparatus |
-
2018
- 2018-12-12 CN CN201811522747.8A patent/CN109857802A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040030703A1 (en) * | 2002-08-12 | 2004-02-12 | International Business Machines Corporation | Method, system, and program for merging log entries from multiple recovery log files |
| CN107291787A (en) * | 2016-04-13 | 2017-10-24 | 中兴通讯股份有限公司 | Master/slave data storehouse switching method and apparatus |
| CN106407356A (en) * | 2016-09-07 | 2017-02-15 | 网易（杭州）网络有限公司 | Data backup method and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110471774A (en) * | 2019-06-28 | 2019-11-19 | 苏宁云计算有限公司 | A kind of data processing method and device based on unified task schedule |
| CN112395157A (en) * | 2020-11-13 | 2021-02-23 | 广州至真信息科技有限公司 | Audit log obtaining method and device, computer equipment and storage medium |
| CN112395157B (en) * | 2020-11-13 | 2023-08-08 | 广州至真信息科技有限公司 | Audit log acquisition method and device, computer equipment and storage medium |
| CN114666753A (en) * | 2020-12-04 | 2022-06-24 | 中国移动通信集团广东有限公司 | Signaling stream data processing method, position information service method and electronic equipment |
| CN114666753B (en) * | 2020-12-04 | 2023-08-15 | 中国移动通信集团广东有限公司 | Signaling stream data processing method, position information service method and electronic equipment |
| WO2022134876A1 (en) * | 2020-12-24 | 2022-06-30 | 中兴通讯股份有限公司 | Data synchronization method and apparatus, and electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109857802A (en) | Daily record data synchronous method, device, equipment and computer readable storage medium | |
| CN103677539B (en) | Interface method of adjustment and device | |
| RU2614538C2 (en) | Tracking data usage under a schematized data plan | |
| CN107040578A (en) | Method of data synchronization, device and system | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN101557427A (en) | Method for providing diffluent information and realizing the diffluence of clients, system and server thereof | |
| JP2010512126A (en) | Apparatus and method for providing and providing an indication of communication events on a map | |
| CN107092628B (en) | Time series data processing method and device | |
| CN103067297A (en) | Dynamic load balancing method and device based on resource consumption prediction | |
| CN102622791A (en) | Network sign-in system and method | |
| CN110610376A (en) | Behavior data response method and device, computer equipment and storage medium | |
| CN104216698A (en) | Webpage registration method and relative device | |
| CN109656783A (en) | System platform monitoring method and device | |
| CN109241031B (en) | Model generation method, model using method, device, system and storage medium | |
| CN113422808B (en) | Internet of things platform HTTP information pushing method, system, device and medium | |
| CN102340791A (en) | System used for realizing data consistency and method thereof | |
| CN109474902A (en) | Message message accounting, method, terminal and system | |
| CN109582439A (en) | DCN dispositions method, device, equipment and computer readable storage medium | |
| US20160103924A1 (en) | Method and apparatus for acquiring information digest | |
| CN106055637B (en) | A kind of caching data processing method and server | |
| CN108647256A (en) | File synchronisation method, device, computer installation and storage medium | |
| CN108182122B (en) | Application update reminding method and device, storage medium and server | |
| US11115464B2 (en) | Server apparatus, information processing method, and computer program product | |
| CN104410743A (en) | Contact information display method and contact information display device | |
| CN109919669A (en) | Dimension management method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190607 |
|
| RJ01 | Rejection of invention patent application after publication |