CN104462483A - Data synchronizing method, device and system - Google Patents
Data synchronizing method, device and system Download PDFInfo
- Publication number
- CN104462483A CN104462483A CN201410797462.0A CN201410797462A CN104462483A CN 104462483 A CN104462483 A CN 104462483A CN 201410797462 A CN201410797462 A CN 201410797462A CN 104462483 A CN104462483 A CN 104462483A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- database cluster
- cluster
- written
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000012544 monitoring process Methods 0.000 claims description 40
- 238000010586 diagram Methods 0.000 description 7
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data synchronizing method, device and system. The data synchronizing method is used for synchronizing data in a first database cluster to a second database cluster, wherein the first database cluster and the second database cluster are independent of each other. The data synchronizing method includes the steps that a data write operation of the first database cluster is monitored; when it is monitored that data are written in the first database cluster, the written data are read; the data are written in the second database cluster. By means of the technical scheme, data synchronization between the first database cluster and the second database cluster is achieved, the problem that data synchronization among clusters cannot be achieved by an independent cluster is solved, the second database cluster provides data access services to the outside, and thus the pressure of accessed data of the first database cluster is effectively shared.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data synchronization method, a data synchronization device and a data synchronization system.
Background
Mongobb, a distributed document storage database, written in the C + + language, aims to provide an extensible, high-performance data storage solution for world Wide Web (WEB) applications. Mongodb is a high-performance, open-source, schema-less document-type database, which is one of the more popular types in the current non-relational database. It can be used in many scenarios to replace traditional relational databases or key/value storage approaches.
Mongodb only allows 7 nodes to be accommodated in one cluster, and each node receives requests of external data, such as addition, deletion, modification and search requests, and further calculates the nodes of the data and copies thereof distributed in the cluster according to the request content, and is responsible for the summary of the request distribution and the response. With the gradual increase of the traffic volume, the problem that 7 nodes cannot bear the existing request volume occurs.
Disclosure of Invention
In view of the above, the present invention has been made to provide a data synchronization method, apparatus and system that overcome or at least partially solve the above problems.
According to an aspect of the present invention, there is provided a data synchronization method for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the method comprising: monitoring data write operations of the first database cluster; reading the written data when it is monitored that there is data written to the first database cluster; and writing the data to the second database cluster.
Optionally, the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further includes: and monitoring data writing operation of a master database in the first database cluster in real time.
Optionally, the step of monitoring, in real time, data write operations of a master database in the first database cluster further includes: and monitoring whether the parameters representing the write operation in the oplog log file of the main database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
Optionally, the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further includes: and monitoring data writing operation of at least one slave database in the first database cluster in real time.
Optionally, the step of monitoring data write operations of at least one slave database in the first database cluster in real time further comprises: and monitoring whether the parameters which represent the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
Optionally, the second database cluster includes a master database and at least one slave database, and the step of writing the data into the second database cluster further includes: writing the data to the master database of the second database cluster.
Optionally, after writing the data into the master database of the second database cluster, the method further includes: synchronizing the data from the master database of the second database cluster to the at least one slave database of the second database cluster.
Optionally, the first database cluster and the second database cluster are Mongodb clusters.
Optionally, the reading the written data includes: reading the written data through a Mongodb interface.
According to another aspect of the present invention, there is also provided a data synchronization apparatus for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the data synchronization apparatus comprising:
a monitor adapted to monitor data write operations of the first database cluster;
a reader adapted to read data written when it is monitored that there is said data written to said first database cluster; and
a writer adapted to write the data to the second database cluster.
Optionally, the first database cluster comprises a master database and at least one slave database, and the monitor is further adapted to: and monitoring data writing operation of a master database in the first database cluster in real time.
Optionally, the monitor is further adapted to: and monitoring whether the parameters representing the write operation in the oplog log file of the main database are assigned or not in real time, and if so, determining that data are written into the main first database cluster.
Optionally, the first database cluster comprises a master database and at least one slave database, and the monitor is further adapted to: and monitoring data writing operation of at least one slave database in the first database cluster in real time.
Optionally, the monitor is further adapted to: and monitoring whether the parameters which represent the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
Optionally, the second database cluster comprises a master database and at least one slave database, the writer is further adapted to: writing the data to the master database of the second database cluster.
Optionally, the apparatus further comprises:
a synchronizer adapted to synchronize the data from the master database of the second database cluster to the at least one slave database of the second database cluster after the writer writes the data to the master database of the second database cluster.
Optionally, the first database cluster and the second database cluster are Mongodb clusters.
Optionally, the reader is further adapted to: reading the written data through a Mongodb interface.
According to another aspect of the present invention, there is also provided a data synchronization system, which includes a first database cluster, a second database cluster and the data synchronization apparatus.
Optionally, the data synchronization apparatus is independent of the first database cluster and the second database cluster; alternatively, the data synchronization apparatus resides in the first database cluster or the second database cluster.
The technical scheme provided by the invention is applied to at least two database clusters which cannot realize the synchronization among the clusters. When a certain database cluster can not bear the user request, another database cluster needs to carry out cooperative load, but if another database cluster can not realize data synchronization, the purpose of cooperative service can not be realized. To solve the technical problem, an embodiment of the present invention provides a method for synchronizing data across database clusters, which is used to synchronize data in a first database cluster to a second database cluster, that is, monitor a data write operation of the first database cluster, and when it is monitored that there is data written into the first database cluster, read the written data, and then write the written data into the second database cluster, so that the second database cluster provides a data access service to the outside. Therefore, data synchronization of the first database cluster and the second database cluster is achieved, and the problem that data synchronization between the clusters cannot be achieved by the independent database clusters is solved. And after the data of the first database cluster is synchronized to the second database cluster, the second database cluster can provide data access service to the outside, so that the problem that the first database cluster cannot continuously provide the data access service when the access request amount reaches an extreme value which can be borne by the first database cluster in the related art is solved, and the pressure of the first database cluster on the accessed data is effectively shared.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments thereof, taken in conjunction with the accompanying drawings.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 shows a flow diagram of a data synchronization method according to one embodiment of the invention;
FIG. 2 is a schematic diagram of a data synchronization apparatus according to an embodiment of the present invention;
FIG. 3 is a schematic diagram showing a structure of a data synchronization apparatus according to another embodiment of the present invention; and
fig. 4 shows a schematic structural diagram of a data synchronization system according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In order to solve the foregoing technical problem, an embodiment of the present invention provides a data synchronization method for synchronizing data in a first database cluster to a second database cluster, where the first database cluster and the second database cluster are independent of each other, and the database cluster itself cannot implement data synchronization between the two database clusters. It should be noted that, the first and second database clusters in the embodiment of the present invention are not limited in number, and there may be at least one first database cluster and at least one second database cluster. Fig. 1 shows a flow diagram of a data synchronization method according to an embodiment of the invention. As shown in fig. 1, the method includes at least the following steps S102, S104, and S106.
And S102, monitoring data writing operation of the first database cluster.
And step S104, reading the written data when the data is monitored to be written into the first database cluster.
And step S106, writing the written data into a second database cluster.
The technical scheme provided by the invention is applied to at least two database clusters which cannot realize the synchronization among the clusters. When a certain database cluster can not bear the user request, another database cluster needs to carry out cooperative load, but if another database cluster can not realize data synchronization, the purpose of cooperative service can not be realized. To solve the technical problem, an embodiment of the present invention provides a method for synchronizing data across database clusters, which is used to synchronize data in a first database cluster to a second database cluster, that is, monitor a data write operation of the first database cluster, and when it is monitored that there is data written into the first database cluster, read the written data, and then write the written data into the second database cluster, so that the second database cluster provides a data access service to the outside. Therefore, data synchronization of the first database cluster and the second database cluster is achieved, and the problem that data synchronization between the clusters cannot be achieved by the independent database clusters is solved. And after the data of the first database cluster is synchronized to the second database cluster, the second database cluster can provide data access service to the outside, so that the problem that the first database cluster cannot continuously provide the data access service when the access request amount reaches an extreme value which can be borne by the first database cluster in the related art is solved, and the pressure of the first database cluster on the accessed data is effectively shared.
In an embodiment of the present invention, the first database cluster may include a master database and at least one slave database, and thus any one of the master database or the at least one slave database of the first database cluster may be monitored. That is, step S102 above may be implemented to monitor in real time data write operations of the master database in the first database cluster. In a preferred embodiment, whether a parameter indicating a write operation is assigned in an oplog (dispatch log) log file of the master database may be monitored in real time, and if so, it is determined that there is data written to the master database (i.e., data is written to the first database cluster); otherwise, it is determined that no data is written to the master database. For example, monitoring whether an insert parameter or an update parameter in an oplog log file of the main database is assigned in real time, and if so, determining that the monitored existence data is written into the main database; if not, determining that no data is monitored to be written into the main database. In addition, in the case of the non-instantaneous data service, step S102 may also be implemented to monitor, in real time, data write operations of at least one slave database in the first database cluster. In another preferred embodiment, whether the parameter representing the write operation in the oplog log file of the slave database is assigned or not can be monitored in real time, and if so, it is determined that the data is written into the slave database (that is, the data is written into the first database cluster); otherwise, it is determined that no data is written to the slave database. For example, whether insert or update parameters are assigned in an oplog log file of the slave database is monitored in real time, and if yes, it is determined that the monitored existence data is written into the slave database; if not, determining that the slave database is not monitored to have data written.
Further, the second database cluster may include a master database and at least one slave database, and step S106 may be implemented to write data into the master database of the second database cluster. Thereafter, data is synchronized from the master database of the second database cluster to its slave database.
In one embodiment of the invention, the first and second database clusters may be Mongodb clusters. The Mongodb cluster is an independent cluster, and the data synchronization among the clusters cannot be realized by the cluster, and by adopting the technical scheme provided by the invention, the data of the Mongodb first database cluster can be synchronized to the Mongodb second database cluster, so that the second database cluster can provide data access service to the outside. For example, the amount of access requests that the first montodb database cluster can bear is 1000, and if the current amount of access requests reaches 1001, the second montodb database cluster can provide services for the 1001 st access request. For another example, the amount of access requests that the first montodb database cluster can bear is 1000, and if the amount of access requests is 1500, 800 access requests of the first montodb database cluster can be serviced, and the remaining 700 access requests can be serviced by the second montodb database cluster. For another example, the access request amount that the first Mongodb database cluster can bear is 1000, the access request amount that the second Mongodb database cluster a can bear is 1000, and if the access request amount is 2500, at this time, two clusters cannot meet the current requirement, and the clusters need to be added again, then the data of the first Mongodb database cluster can be synchronized to the second Mongodb database cluster B, and the second database cluster B provides services for 500 access requests. It should be noted that the examples are only illustrative and not intended to limit the present invention.
Further, when it is monitored that there is data written into the first database cluster in step S104, reading the written data may read the written data through the montodb interface. In addition, in step S106, the written data is written into the second database cluster, and the written data can be written into the second database cluster through the Mongodb interface.
Based on the same inventive concept, the embodiment of the invention also provides a data synchronization device to realize the data synchronization method.
Fig. 2 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present invention. Referring to fig. 2, the apparatus comprises at least: monitor 210, reader 220, and writer 230.
The functions of the components or devices of the data synchronization apparatus of the embodiment of the present invention and the connection relationship between the components will now be described:
a monitor 210 adapted to monitor data write operations of a first database cluster;
a reader 220, coupled to the monitor 210, adapted to read the written data when it is monitored that there is data written to the first database cluster; and
a writer 230, coupled to the reader 220, is adapted to write data to the second database cluster.
In one embodiment of the present invention, the first database cluster comprises a master database and at least one slave database, and the monitor 210 is further adapted to: data write operations of a master database in a first database cluster are monitored in real time.
In an embodiment of the present invention, the monitor 210 is further adapted to: and monitoring whether the parameters of the write operation are assigned or not in an oplog log file of the main database in real time, and if so, determining that the data are written into the main database (namely, the data are written into the first database cluster).
In one embodiment of the present invention, the first database cluster comprises a master database and at least one slave database, and the monitor 210 is further adapted to: and monitoring data writing operation of at least one slave database in the first database cluster in real time.
In an embodiment of the present invention, the monitor 210 is further adapted to: and monitoring whether the parameters representing the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that the data are written into the slave database (namely, the data are written into the first database cluster).
In one embodiment of the present invention, the second database cluster comprises a master database and at least one slave database, writer 230 is further adapted to: data is written to a master database of the second database cluster.
In an embodiment of the present invention, fig. 3 shows a schematic structural diagram of a data synchronization apparatus according to another embodiment of the present invention. Referring to fig. 3, the apparatus may further include: synchronizer 310, coupled to writer 230, is adapted to synchronize data from the master database of the second database cluster to at least one slave database of the second database cluster after writer 230 writes the data to the master database of the second database cluster. Note that the synchronization operation is not limited to be performed by the synchronizer 310. Since the Mongodb cluster itself also has the function of synchronizing the master database data to its slave databases, the synchronization action can also be done by the Mongodb itself without the synchronizer 310, which is not a limitation of the present invention.
In one embodiment of the invention, the first database cluster and the second database cluster are Mongodb clusters.
In one embodiment of the present invention, the reader 220 is further adapted to: the written data is read through the Mongodb interface.
Based on the data synchronization method and apparatus provided by the foregoing embodiments, based on the same inventive concept, the embodiments of the present invention further provide a data synchronization system, as shown in fig. 4, where the system at least includes: a first database cluster 410, a second database cluster 420, and a data synchronization apparatus 430 (shown in fig. 2).
A data synchronization means 430 adapted to monitor data write operations of the first database cluster 410; when it is monitored that there is data written to the first database cluster 410, the written data is read; writing the written data to a second database cluster 420;
the second database cluster 420 is adapted to receive a data write request from the data synchronization apparatus 430 and perform a data write operation.
In one embodiment of the invention, the data synchronization apparatus 430 is independent of the first database cluster 410 and the second database cluster 420; or,
the data synchronization apparatus 430 resides in the first database cluster 410 or the second database cluster 420.
In an embodiment of the present invention, when a data access request is received, a preset offloading policy may be employed to offload the data access request, and part of the data access request is accessed by using the first database cluster 410, and part of the data access request is accessed by using the second database cluster 420. The offloading policy may include bandwidth load balancing, traffic load balancing, or request average access delay, etc., and the present invention is not limited thereto.
According to any one or a combination of the above preferred embodiments, the following advantages can be achieved by the embodiments of the present invention:
the technical scheme provided by the invention is applied to at least two database clusters which cannot realize the synchronization among the clusters. When a certain database cluster can not bear the user request, another database cluster needs to carry out cooperative load, but if another database cluster can not realize data synchronization, the purpose of cooperative service can not be realized. To solve the technical problem, an embodiment of the present invention provides a method for synchronizing data across database clusters, which is used to synchronize data in a first database cluster to a second database cluster, that is, monitor a data write operation of the first database cluster, and when it is monitored that there is data written into the first database cluster, read the written data, and then write the written data into the second database cluster, so that the second database cluster provides a data access service to the outside. Therefore, data synchronization of the first database cluster and the second database cluster is achieved, and the problem that data synchronization between the clusters cannot be achieved by the independent database clusters is solved. And after the data of the first database cluster is synchronized to the second database cluster, the second database cluster can provide data access service to the outside, so that the problem that the first database cluster cannot continuously provide the data access service when the access request amount reaches an extreme value which can be borne by the first database cluster in the related art is solved, and the pressure of the first database cluster on the accessed data is effectively shared.
The invention also discloses:
a1, a data synchronization method for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the method comprising:
monitoring data write operations of the first database cluster;
reading the written data when it is monitored that there is data written to the first database cluster; and
writing the data to the second database cluster.
A2, the method of A1, wherein the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further includes:
and monitoring data writing operation of a master database in the first database cluster in real time.
A3, the method of A2, wherein the step of monitoring in real time data write operations of master databases in the first database cluster further comprises:
and monitoring whether the parameters representing the write operation in the oplog log file of the main database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
A4, the method of A1, wherein the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further includes:
and monitoring data writing operation of at least one slave database in the first database cluster in real time.
A5, the method of A4, wherein the step of monitoring data write operations of at least one slave database in the first database cluster in real time further comprises:
and monitoring whether the parameters which represent the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
A6, the method of A1, wherein the second database cluster includes a master database and at least one slave database, and the step of writing the data to the second database cluster further includes:
writing the data to the master database of the second database cluster.
A7, the method of A6, wherein after writing the data to the master database of the second cluster of databases, further comprising:
synchronizing the data from the master database of the second database cluster to the at least one slave database of the second database cluster.
A8, the method according to any of A1-A7, wherein the first and second database clusters are Mongdb clusters.
A9, the method of A8, wherein the reading the written data includes:
reading the written data through a Mongodb interface.
B10, a data synchronization apparatus for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the data synchronization apparatus comprising:
a monitor adapted to monitor data write operations of the first database cluster;
a reader adapted to read data written when it is monitored that there is said data written to said first database cluster; and
a writer adapted to write the data to the second database cluster.
B11, the apparatus of B10, wherein the first database cluster includes a master database and at least one slave database, the monitor further adapted to:
and monitoring data writing operation of a master database in the first database cluster in real time.
B12, the apparatus according to B11, wherein the monitor is further adapted to:
and monitoring whether the parameters representing the write operation in the oplog log file of the main database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
B13, the apparatus of B10, wherein the first database cluster includes a master database and at least one slave database, the monitor further adapted to:
and monitoring data writing operation of at least one slave database in the first database cluster in real time.
B14, the apparatus according to B13, wherein the monitor is further adapted to:
and monitoring whether the parameters which represent the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
The apparatus of B15, the apparatus of B10, wherein the second cluster of databases comprises a master database and at least one slave database, the writer further adapted to:
writing the data to the master database of the second database cluster.
B16, the device according to B15, wherein further comprising:
a synchronizer adapted to synchronize the data from the master database of the second database cluster to the at least one slave database of the second database cluster after the writer writes the data to the master database of the second database cluster.
The apparatus of any of B17, B10-B16, wherein the first database cluster and the second database cluster are Mongodb clusters.
B18, the apparatus according to B17, wherein the reader is further adapted to:
reading the written data through a Mongodb interface.
C19, a data synchronization system, comprising a first database cluster, a second database cluster and the data synchronization device of any one of B10-B18.
C20, the system of C19, wherein the data synchronization apparatus is independent of the first and second database clusters; or,
the data synchronization apparatus resides in the first database cluster or the second database cluster.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a data synchronization apparatus and system in accordance with embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Thus, it should be appreciated by those skilled in the art that while a number of exemplary embodiments of the invention have been illustrated and described in detail herein, many other variations or modifications consistent with the principles of the invention may be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Accordingly, the scope of the invention should be understood and interpreted to cover all such other variations or modifications.
Claims (10)
1. A data synchronization method for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the method comprising:
monitoring data write operations of the first database cluster;
reading the written data when it is monitored that there is data written to the first database cluster; and
writing the data to the second database cluster.
2. The method of claim 1, wherein the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further comprises:
and monitoring data writing operation of a master database in the first database cluster in real time.
3. The method of claim 2, wherein the step of monitoring in real time data write operations of a master database in the first database cluster further comprises:
and monitoring whether the parameters representing the write operation in the oplog log file of the main database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
4. The method of claim 1, wherein the first database cluster includes a master database and at least one slave database, and the step of monitoring data write operations of the first database cluster further comprises:
and monitoring data writing operation of at least one slave database in the first database cluster in real time.
5. The method of claim 4, wherein the step of monitoring in real time data write operations of at least one slave database in the first database cluster further comprises:
and monitoring whether the parameters which represent the write operation in the oplog log file of the at least one slave database are assigned or not in real time, and if so, determining that data are written into the first database cluster.
6. The method of claim 1, wherein the second database cluster includes a master database and at least one slave database, and the step of writing the data to the second database cluster further comprises:
writing the data to the master database of the second database cluster.
7. The method of claim 6, wherein writing the data to the master database of the second database cluster further comprises:
synchronizing the data from the master database of the second database cluster to the at least one slave database of the second database cluster.
8. The method of any of claims 1-7, wherein the first database cluster and the second database cluster are Mongdb clusters.
9. A data synchronization apparatus for synchronizing data in a first database cluster to a second database cluster, the first database cluster and the second database cluster being independent of each other, the data synchronization apparatus comprising:
a monitor adapted to monitor data write operations of the first database cluster;
a reader adapted to read data written when it is monitored that there is said data written to said first database cluster; and
a writer adapted to write the data to the second database cluster.
10. A data synchronization system comprising a first database cluster, a second database cluster, and the data synchronization apparatus of claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410797462.0A CN104462483A (en) | 2014-12-18 | 2014-12-18 | Data synchronizing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410797462.0A CN104462483A (en) | 2014-12-18 | 2014-12-18 | Data synchronizing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104462483A true CN104462483A (en) | 2015-03-25 |
Family
ID=52908518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410797462.0A Pending CN104462483A (en) | 2014-12-18 | 2014-12-18 | Data synchronizing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104462483A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN106331063A (en) * | 2016-08-15 | 2017-01-11 | 成都轻车快马网络科技有限公司 | Database deployment method |
| CN107124469A (en) * | 2017-06-07 | 2017-09-01 | 郑州云海信息技术有限公司 | A kind of clustered node communication means and system |
| CN107145562A (en) * | 2017-05-02 | 2017-09-08 | 北京奇艺世纪科技有限公司 | A kind of method of data synchronization, apparatus and system |
| CN107370809A (en) * | 2017-07-13 | 2017-11-21 | 广州市百果园信息技术有限公司 | Method of data synchronization and data search system |
| CN107451245A (en) * | 2017-07-27 | 2017-12-08 | 联想(北京)有限公司 | Method of data synchronization and server cluster |
| CN107644035A (en) * | 2016-07-21 | 2018-01-30 | 百度在线网络技术（北京）有限公司 | A kind of Database Systems and its dispositions method |
| CN108881379A (en) * | 2018-05-03 | 2018-11-23 | 网宿科技股份有限公司 | The synchronous method and apparatus of data between a kind of server cluster |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101854400A (en) * | 2010-06-09 | 2010-10-06 | 中兴通讯股份有限公司 | Database synchronization deployment and monitoring method and device |
| CN102158540A (en) * | 2011-02-18 | 2011-08-17 | 广州从兴电子开发有限公司 | System and method for realizing distributed database |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN102750283A (en) * | 2011-04-20 | 2012-10-24 | 阿里巴巴集团控股有限公司 | Massive data synchronization system and method |
| CN103150304A (en) * | 2011-12-06 | 2013-06-12 | 郑红云 | Cloud database system |
| US8548945B2 (en) * | 2011-09-23 | 2013-10-01 | International Business Machines Corporation | Database caching utilizing asynchronous log-based replication |
| CN103747060A (en) * | 2013-12-26 | 2014-04-23 | 惠州华阳通用电子有限公司 | Distributed monitor system and method based on streaming media service cluster |
-
2014
- 2014-12-18 CN CN201410797462.0A patent/CN104462483A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101854400A (en) * | 2010-06-09 | 2010-10-06 | 中兴通讯股份有限公司 | Database synchronization deployment and monitoring method and device |
| CN102158540A (en) * | 2011-02-18 | 2011-08-17 | 广州从兴电子开发有限公司 | System and method for realizing distributed database |
| CN102750283A (en) * | 2011-04-20 | 2012-10-24 | 阿里巴巴集团控股有限公司 | Massive data synchronization system and method |
| US8548945B2 (en) * | 2011-09-23 | 2013-10-01 | International Business Machines Corporation | Database caching utilizing asynchronous log-based replication |
| CN103150304A (en) * | 2011-12-06 | 2013-06-12 | 郑红云 | Cloud database system |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN103747060A (en) * | 2013-12-26 | 2014-04-23 | 惠州华阳通用电子有限公司 | Distributed monitor system and method based on streaming media service cluster |
Non-Patent Citations (1)
| Title |
|---|
| 白坤娟: "认证系统异构数据库数据同步方法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105912628A (en) * | 2016-04-07 | 2016-08-31 | 北京奇虎科技有限公司 | Synchronization method and device for master database and slave database |
| CN105912628B (en) * | 2016-04-07 | 2019-05-28 | 北京奇虎科技有限公司 | The synchronous method and device of master-slave database |
| CN107644035A (en) * | 2016-07-21 | 2018-01-30 | 百度在线网络技术（北京）有限公司 | A kind of Database Systems and its dispositions method |
| CN106331063B (en) * | 2016-08-15 | 2019-04-09 | 成都轻车快马网络科技有限公司 | Database portion arranging method |
| CN106331063A (en) * | 2016-08-15 | 2017-01-11 | 成都轻车快马网络科技有限公司 | Database deployment method |
| CN107145562A (en) * | 2017-05-02 | 2017-09-08 | 北京奇艺世纪科技有限公司 | A kind of method of data synchronization, apparatus and system |
| CN107145562B (en) * | 2017-05-02 | 2020-07-28 | 北京奇艺世纪科技有限公司 | Data synchronization method, device and system |
| CN107124469A (en) * | 2017-06-07 | 2017-09-01 | 郑州云海信息技术有限公司 | A kind of clustered node communication means and system |
| CN107370809A (en) * | 2017-07-13 | 2017-11-21 | 广州市百果园信息技术有限公司 | Method of data synchronization and data search system |
| CN107451245A (en) * | 2017-07-27 | 2017-12-08 | 联想(北京)有限公司 | Method of data synchronization and server cluster |
| CN108881379A (en) * | 2018-05-03 | 2018-11-23 | 网宿科技股份有限公司 | The synchronous method and apparatus of data between a kind of server cluster |
| CN112667698A (en) * | 2021-01-04 | 2021-04-16 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
| CN112667698B (en) * | 2021-01-04 | 2023-02-28 | 山西云媒体发展有限公司 | MongoDB data synchronization method based on converged media platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104462483A (en) | Data synchronizing method, device and system | |
| JP6745884B2 (en) | Data synchronization method, device and system | |
| US9727268B2 (en) | Management of storage in a storage network | |
| US8874700B2 (en) | Optimizing storage of data files | |
| KR101932395B1 (en) | Activity continuation between electronic devices | |
| US20140289382A1 (en) | Utilizing version vectors across server and client changes to determine device usage by type, app, and time of day | |
| US11245774B2 (en) | Cache storage for streaming data | |
| CN106503158B (en) | Data synchronization method and device | |
| CN111475483B (en) | Database migration method and device and computing equipment | |
| CN110083660A (en) | A kind of method, apparatus of synchrodata, medium and electronic equipment | |
| US10379983B2 (en) | Simulation device and distribution simulation system | |
| CN107391033B (en) | Data migration method and device, computing equipment and computer storage medium | |
| CN104158897A (en) | Updating method of file layout in distributed file system | |
| CN109145053B (en) | Data processing method and device, client and server | |
| US9716666B2 (en) | Process cage providing attraction to distributed storage | |
| JP6940645B2 (en) | Replica processing methods, nodes, storage systems, servers and readable storage media | |
| CN105988941B (en) | Cache data processing method and device | |
| CN106331048A (en) | Data processing method, apparatus and system, and mobile device | |
| EP3178002A1 (en) | Data backup to and restore from trusted devices in close physical proximity | |
| CN106899630B (en) | Thumbnail display method and device for pictures in network disk | |
| US10248316B1 (en) | Method to pass application knowledge to a storage array and optimize block level operations | |
| CN104598396A (en) | System and method for ensuring caching instantaneity and caching consistency of mass data | |
| US20170344432A1 (en) | Data synchronization method and device without redundant replication | |
| CN110728379A (en) | Configuration information management method and device, electronic equipment and non-transitory storage medium | |
| CN105389368A (en) | Method for managing metadata of database cluster of MPP architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150325 |