CN102982085B - Data mover system and method - Google Patents
Data mover system and method Download PDFInfo
- Publication number
- CN102982085B CN102982085B CN201210429724.9A CN201210429724A CN102982085B CN 102982085 B CN102982085 B CN 102982085B CN 201210429724 A CN201210429724 A CN 201210429724A CN 102982085 B CN102982085 B CN 102982085B
- Authority
- CN
- China
- Prior art keywords
- data
- server cluster
- data server
- time point
- write
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data mover system and method, for carrying out Data Migration between data server cluster, data are the target data related to the business to be migrated, and the method includes that the target data write before first time point and the first time point that will be stored in the first data server cluster imports the second data server cluster；The Operation Log that acquisition is associated with the target data that the first data server cluster is write after first time point；Target data in second data server cluster is updated according to the Operation Log for obtaining；Detect whether the target data of the first data server cluster and the second data server cluster is synchronous；And after the target data synchronization of the first data server cluster and the second data server cluster, the entry address for connecting data server is changed to the entry address of the second data server cluster by the entry address of the first data server cluster.
Description
Technical field
The present invention relates to technical field of data storage, and in particular to a kind of data mover system and method.
Background technology
MongoDB（Data Base, database）It is a product between relational database and non-relational database
Product, are that function is most abundant in the middle of non-relational database, are most like relational database.The data structure that he supports is very loose, because
This can store more complicated data type.Due to MongoDB better performances in itself, therefore early stage business development, open
Hair personnel can be by multiple small service deployments on the small Mongodb clusters that an a small amount of server is constituted.When business gradually
Development, visit capacity is increasing, such as the visit capacity rapid development of original certain or certain several business, forms large-scale industry
Business, and then cause the small cluster that a small amount of server is constituted cannot to meet business demand, now it is accomplished by considering how to increase
Adding system capacity solves performance issue.
The first current solution is, based on the dynamic expansion that Mongodb is supported in itself, can be simply by increasing
Plus data server improves performance, therefore can be by directly to the Mongodb collection being made up of low volume data server at present
Mongod servers are added in group to solve performance issue.By increasing new Mongod servers, it is changed into more number of units according to clothes
The cluster of business device, the performance of mongodb can substantially linear lifting.But, this solution also brings along side effect.Because existing
It is multiple business services that some data server clusters are, and possible existing small-scale business in multiple business, is also had
The large-scale business that rapid growth is got up, and then in the access process of Mongod servers, large-scale business may
The access resource to Mongod servers can be for a long time occupied, and small-scale business certainly will fight for only large-scale business, most
These small-scale business are caused to be subject to influenceing for the large-scale business of rapid growth eventually.
In order to avoid the side effect of above-mentioned the first solution, second solution has been engendered.Considering will
The business that business scale is expanded, moves on a brand-new mongodb cluster from original mongodb clusters.For compared with
Large-scale business, using single new cluster, and no longer shares Mongod servers with other business.So, it is original many
The individual less business of scale is still in original data server cluster, and fast development is used alone into fairly large business
Brand-new data server cluster, then, fairly large business would not seize server with the business of small-scale again
Access resource.
In the transition process for taking such scheme, all write-ins to mongod servers are cut off by the business first and is grasped
Make, then back up out by existing mongodb database informations, imported into new data server cluster.Finally, inciting somebody to action
After the data of business are switched to new data server cluster completely, write operation of the business to mongod servers is then turned on.
Due to needing to cut off all write services of business in handoff procedure, and when data volume is larger, backup migration is shut down
Process may may require that several hours, therefore influence very much the normal of related service within this period for shutting down backup migration
Operation, influence provides the user normal service.Similarly, other non-mongodb data server cluster applied environment
Under, similarly there are Similar Problems.
The content of the invention
In view of the above problems, it is proposed that the present invention so as to provide one kind overcome above mentioned problem or at least in part solve on
State the data mover system and method for problem.
According to one aspect of the present invention, there is provided a kind of data mover system, between data server cluster
Data Migration is carried out, the system at least includes that the first data server cluster and the second data server cluster and migration set
Standby, data are the target data related to the business to be migrated, and the migration equipment includes：Primary data import modul, is matched somebody with somebody
It is set to the target data write before first time point and the first time point that will be stored in the first data server cluster
Import the second data server cluster；Synchronization module, is configured as obtaining and the first data clothes is write after first time point
The Operation Log that the target data of device cluster of being engaged in is associated, Operation Log is at least included in the first data server cluster of write-in
The content of target data；Update module, the Operation Log for being configured as being obtained according to synchronization module updates the second data server
Target data in cluster.
Optionally, primary data import modul includes：First primary data imports submodule, is configured as the first data
The target data write before the first time point stored in server cluster and the first time point imports a storage medium；The
Two primary datas import submodule, and the target data for being configured as to be directed into storage medium imports the second data server collection
Group.
Optionally, wherein the first primary data imports submodule is configured to mongodump by the first data, services
The target data write before the first time point stored in device cluster and the first time point backs up to a storage medium；And
Second primary data imports submodule and is configured to the target data that mongorestore will be backed up in storage medium to be led
Enter the second data server cluster.
Optionally, the first data server cluster includes master data server and some from data server, and migration sets
It is standby also to include：Processing module is disabled, is configured as after first time point stopping in the first data server cluster first
From the write operation of data server；And primary data import modul, it is configured as the of the first data server cluster
The target data write before one first time point stored from data server and the first time point imports the second data
Server cluster.
Optionally, also include：Synchronous detection module, is configured as the first data server cluster of detection and the second data clothes
Whether the target data of business device cluster is synchronous；Address update module, is configured as detecting the first number in synchronous detection module
After according to the target data synchronization of server cluster and the second data server cluster, the entrance ground of data server will be connected
Location is changed to the entry address of the second data server cluster by the entry address of the first data server cluster.
Optionally, Operation Log also includes one or more in following information：Write the timestamp of data；When write-in
When data are the renewals to legacy data, the old value before data renewal.
Optionally, the first data server cluster is a MongoDB clusters, and the second data server cluster is second
MongoDB clusters, Operation Log is the oplog in MongoDB.
According to another embodiment of the invention, there is provided a kind of data migration method, in data server cluster
Between carry out Data Migration, data are the target data related to the business to be migrated, including：By the first data server cluster
The target data write before first time point and the first time point of middle storage imports the second data server cluster；Obtain
Obtain the Operation Log being associated with the target data that the first data server cluster is write after first time point, Operation Log
At least include the content of the target data in the first data server cluster of write-in；Second is updated according to the Operation Log for obtaining to count
According to the target data in server cluster；Detect first data server cluster and second data server cluster
Whether target data is synchronous；And, in first data server cluster and the mesh of second data server cluster
After mark data synchronization, the entry address of data server will be connected by the entry address of first data server cluster
It is changed to the entry address of second data server cluster.
Data mover system of the invention and method, on the one hand backup by way of by certain time point before
Target data directly backs up to new data server cluster, on the other hand will be write after first time point using Operation Log
The target data of legacy data server cluster is synchronously written into the second data server cluster so that make new and old data server collection
Group realizes the synchronization of target data substantially, so it is follow-up be migrated business and can be directly connected to new data service device cluster carry out
The write-in of data and read, the business to be migrated need not be stopped in the process, thus solving existing must shut down
Backup could realize the problem of business datum migration, achieve the energy in the case where not influenceing to be migrated business external normal service
The beneficial effect of enough finishing service Data Migrations.
Described above is only the general introduction of technical solution of the present invention, in order to better understand technological means of the invention,
And can be practiced according to the content of specification, and in order to allow the above and other objects of the present invention, feature and advantage can
Become apparent, below especially exemplified by specific embodiment of the invention.
Brief description of the drawings
By reading the detailed description of hereafter preferred embodiment, various other advantages and benefit is common for this area
Technical staff will be clear understanding.Accompanying drawing is only used for showing the purpose of preferred embodiment, and is not considered as to the present invention
Limitation.And in whole accompanying drawing, identical part is denoted by the same reference numerals.In the accompanying drawings：
Fig. 1 show it is according to an embodiment of the invention for carrying out Data Migration between data server cluster
One system schematic;
Fig. 2 show it is according to an embodiment of the invention for carrying out Data Migration between data server cluster
Two system schematic diagram;And
Fig. 3 shows the flow chart of data migration method according to an embodiment of the invention.
Specific embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in accompanying drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
Limited.Conversely, there is provided these embodiments are able to be best understood from the disclosure, and can be by the scope of the present disclosure
Complete conveys to those skilled in the art.
Fig. 1 is referred to, it is to be moved for carrying out data between data server cluster according to one embodiment of the invention
The first system schematic diagram of shifting.The system include for Data Migration is carried out between data server cluster migration equipment 100,
First data server cluster 200 and the second data server cluster 300.Migration equipment 100 includes primary data import modul
102nd, synchronization module 104 and update module 106.First data server cluster 200 includes multiple data servers, in figure only
The data server 204 of the first data server therein 202 and second is show schematically, can also basis in practical application
Need comprising more data servers, the present invention is not restricted to this.Similar, the second data server cluster 300
Including multiple data servers, representatively illustrated in figure the 3rd data server 302, the 4th data server 304 and
5th data server 306.
Typically, data backup each other between each data server in each data server cluster, it is general multiple
It is master data server that one is had in data server, and remaining is that application server 400 is in most cases from data server
It is directly that remaining does not typically receive application server 400 directly from data server to data are write in master data server
Data write, but from master data server backup data.With reference to each part to the processing procedure of data, and each part
Between incidence relation be described in detail.Be that follow-up narration is convenient, it would be desirable to from the first data server cluster 200 migrate to
The business datum of the second data server cluster 300, target data referred to as related to the business to be migrated.
In one embodiment, before target data is migrated, application server 400 is all the number that will migrate business
According to i.e. target data writes the first data server cluster 200 at any time.In such as the first data server cluster 200 first
All be stored with the target data having been written into the data server 204 of data server 202 and second.And then, it is being ready for
During Data Migration, selection one first has been the time point of past tense, referred to as first time point, is then led by primary data
Enter module 102 and application server 400 is written to the first data server cluster in the first time point and before the very first time
Target data in 200, imported into the second data server 300.Subsequently in order to more easily illustrate, with the first data, services
The second data server in device 200 is main data server, the first data server is to be said from as a example by data server
It is bright.
Specifically, first, primary data export module 102 is from the multiple data in the first data server cluster 200
A data server is selected in server carries out data derivation, generally, one can be selected to carry out data from data server
Derive, during derived from data, not influence the follow-up write-in to master data server of application server 400, for example,
Being selected as from the first data server 202 of data server carries out data derivation.Further, since application server 400 is not
Data directly are write to as from first data server 202 of data server, but in application server 400 to as master
After the write-in data of second data server 204 of data server, the first data server 202 is again from the second data server
204 obtain data, and due to writing the from before first time point derived from the first data server 202 and first time point
The data volume of the target data of one data server cluster 200 may than larger, therefore, in order to preferably avoid in the first data
Carrying out data in server 202 simultaneously derives the various fortuitous events being likely to occur with data write-in, and migration equipment 100 can be wrapped
Deactivation processing module is included, the write operation for can first stop the first data server 202 after first time point, then
Just proceed by the derivation operation of data.Treat that primary data import modul 102 successfully derives first from the first data server 202
Write before time point and the very first time after the data of the first data server cluster 200, can again recover the first data clothes
The write operation of business device 202.It should be noted that first stopping the first data server above in connection with before target data is derived
The example of 202 write operation is only optional, can not stopped from the second data while target data is derived completely
The synchrodata of server 204 is to the first data server 202.
Then, primary data import modul 102 will be stored in first time point and first in the first data server 202
The target data write before time point imported into the second data server cluster 300.For example, primary data import modul 102
Specifically include the first primary data and import submodule and the second primary data importing submodule, the first primary data imports submodule
The target data write before first time point and first time point is first taken out from the first data server 202, and by mesh
Mark data import a storage medium, such as in disk（For example, can be with document form）.Subsequent second primary data imports submodule
Each data server that the target data of the storage medium is imported in the second data server cluster 300 will have been imported again（302-
306）.During target data is imported the second data server cluster 300 by primary data import modul 102, Ke Yixian
By target data import the second data server cluster 300 in master data server, then other from data server again from
Master data server obtains target data, master data server in such second data server cluster 300 and from data clothes
Business device, i.e. the 3rd data server 302, the 4th data server 304 and the 5th data server 306 all have successfully been obtained first
The target data of the first data server cluster 100 is write before time point and first time point.
The first data server collection was write before primary data import modul 102 is by first time point and first time point
Group 200 target data successfully import the second data server cluster 300 during and successfully import after, application server
400 do not stop being write to the second data server 204 as master data server in the first data server cluster 200
Enter data, therefore after first time point, still there is target data to be written into the first data server cluster 200.Specifically
For, the second data server 204 is first employed the target data after the write-in first time point of server 400, subsequent first
The target data that data server 202 can write from after the synchronization gain first time point of the second data server 204 again.
And, when application server 400 often writes a target data to the second data server 204, based on data
The characteristic of server cluster itself, data server cluster all can simultaneously produce an operation being associated with the target data day
Will (oplog, operation log), records the content of the specific data of each write-in in the Operation Log.And, one
As also record in Operation Log each write-in data temporal information, i.e. timestamp, if the data of write-in are to former
The renewal of data, then not only can record data update after new value, but also can record data update before old value.For changing
It, according to Operation Log, it is possible to know application server 400 in which time, be specifically written with the data of which content.
And then, the synchronization module 104 in migration equipment 100 can obtain and the first data are write after first time point
The associated Operation Log of the target data of server cluster 200.Specifically, because application server 400 is to the first number
During according to the write-in target data of server cluster 200, data are write one by one more, accordingly, can also be produced one by one
Raw Operation Log, and Operation Log includes writing the timestamp of data.Therefore optional, synchronization module 104 can be real-time
Go to obtain the Operation Log being associated with the target data for writing the first data server cluster 200 after first time point.
It is of course also possible to need not go to obtain Operation Log in real time, but go to obtain once-through operation daily record at interval of certain hour, but,
In order to realize two synchronizations as early as possible of data server cluster as early as possible, the time interval that synchronization module 104 obtains Operation Log is use up
Amount is short.
Then, synchronization module 104 will obtain and the first data server cluster 200 write after first time point
The associated Operation Log of target data is supplied to update module 106.Update module 106 is obtained and write with after first time point
After entering the associated Operation Log of data of the first data server cluster 200, it is possible to according to the Operation Log for being obtained
Update the target data in the second data server cluster 300.Carried before, application service had been comprised at least in Operation Log
Device 400 writes the content of the specific data of the first data server cluster 200 every time after first time point, therefore updates mould
Block 106 can be according to corresponding target data in every Operation Log second data server cluster 300 of renewal so that the second number
The target data that the first data server cluster 200 is write after first time point is also stored according to server cluster 300.
Because the data more new content that wall scroll Operation Log is related to is little, therefore update module 106 can very timely
The target data for finishing and being associated in the second data server cluster 300 is updated according to Operation Log, and then, substantially can be with
Reach application server 400 after first time point often to the first data server cluster 200 write a target data or
Same target data just can be write the second data server cluster by several target datas, update module 106 accordingly
300, so, after the very of short duration time, the second data server 300 can just reach and the first data server 200
In consistent, the synchronous purpose of target data.
So far, related to business the to be migrated target data for being stored in the first data server cluster 200 is complete
Portion moves to the second data server cluster 300.
In another embodiment, after being migrated successfully for target data, subsequent applications server 400 can directly to
Two data server cluster 300 writes the data related to business to be migrated, and synchronous inspection can also be included in equipment 100 is migrated
Survey module and address update module.Specifically, update module 106 obtained according to synchronization module 104 with the very first time
Associated Operation Log the second data server collection of renewal of the target data of the first data server cluster 200 is write after point
After target data in group 300, synchronous detection module can be to the first data server cluster 200 and the second data server
Whether target data in cluster 300 is contrasted, so as to detect the target data in two data server clusters
Synchronous success.Detect whether synchronous success, except may be referred to the target data in two data server clusters itself in
Hold, the Operation Log that be associated with target data can also be referred to because in Operation Log it is general have write target data when
Between stamp and the auxiliary information such as the forward and backward value of updating target data, therefore, synchronous detection module can also be with reference to these auxiliary letters
Whether breath, more rapidly, accurately detect two data server clusters successful synchronization.
Judge that the first data server cluster 200 and the second data server cluster 300 are same in synchronous detection module
After walking successfully, can be with the two data server clusters synchronized success of Notify Address update module, and then address updates mould
Application server 400 can just be connected the entry address of data server by the entrance of the first data server cluster 200 by block
Address change is the entry address of the second data server cluster 300.Subsequent applications server 400 to data if necessary to take again
The business device cluster write-in data related to business to be migrated read the related data of the business, will be because of connection data clothes
The entry address of business device has been changed to the entry address of the second data server cluster, directly to the second data server cluster
300 write-ins and reading for carrying out data.At this time, it may be necessary to the business being migrated has succeeded from the first data server cluster 200
Migrate to the second data server cluster 300.Then, number related to the business that is migrated in the first data server cluster 200
According to can just delete.
It should be noted that migration equipment 100 is during implementing, can be independently of the first data server cluster
200 and second data server cluster 300 and be implemented separately, it is also possible to be placed in certain data server cluster, such as the second number
It is achieved in data management server according to server cluster 300.The data management server can be current many data
Server all having in server cluster, taking on cluster management role, such as the mangos in MongoDB clusters.
In one embodiment, the first data server cluster 200 of the first system is a MongoDB clusters above
500, the second data server cluster 300 is the 2nd MongoDB clusters 600, and migration equipment 100 therein is placed in second
It is achieved in the mangos of MongoDB clusters.Fig. 2 is specifically referred to, it is for counting according to one embodiment of the invention
According to the second system schematic diagram that Data Migration is carried out between server cluster.The second system can be understood as
System is applied to the concrete application example of the such data server clusters of MongoDB, therefore in second system first
MongoDB clusters 500 duplicate with the first data server cluster 200 in the first system, similar, and in second system second
MongoDB clusters 600 duplicate with the second data server cluster 300 in the first system, and the primary data in second system is led
Enter module 702 to be duplicated with the primary data import modul 102 in the first system, the synchronization module 704 and first in second system
Synchronization module 104 in system duplicates, and the update module 106 in update module 706 in second system and the first system
It is identical, thus to second system in implementing for each part repeat no more substantially, may be referred to be associated in the first system
The specific implementation of part.It is explained only for the individual part for being related to specific MongoDB cluster features.
Such as, the first primary data importing submodule in primary data import modul 702 can be first passed through and call mongo
A kind of backup tool mongodump for carrying, the first time point and first time point that will be stored in a mongod clusters 500
The target data for writing backs up to disk before on, a data file is generated.Then in primary data import modul 702
Two primary data submodules reuse a kind of recovery instrument mongorestore that mongo carries and the data file are imported into
In two mongod clusters 600.The Operation Log that synchronization module 704 is obtained is specifically the oplog in MongoDB, such as one tool
Body oplog example contents are as follows：
{″ts″:{″t″:1339660240000,″i″:8},
″h″:NumberLong(″-7936072258265513667″),″op″:″i″,″ns″:″test.method″,
″o″:{″_id″:″testid″,″v″:”test”}}
Wherein, what " ts " was recorded is the timestamp of the operation；What " op " was recorded is the type of the operation, such as type " i "
It is shown to be insertion operation；What " h " was recorded is the cryptographic Hash of this oplog, and what " ns " was recorded is the NameSpace of the operation；“o”
Record is file content, i.e., the content of the data of specific write-in.
Be can be seen that in the oplog of MongoDB clusters not only comprising the target of write-in from an above-mentioned oplog example
The content of data, but also other auxiliary informations such as including timestamp, therefore update module 706 can be according to synchronization module 704
The Operation Log that is associated of target data for obtaining and a MongoDB clusters 500 being write after first time point, renewal the
The content of the target data in two MongoDB clusters 600.
It will be understood by those skilled in the art that in the distributed data-storage system of other non-MongoDB clusters, for example
In other distributed data-storage systems such as Cassandra, there is also and need to carry out Data Migration between data server cluster
Similar Problems, and with the Operation Log similar with the Operation Log of MongoDB clusters, therefore technical side of the invention
Case is applicable not only to the Data Migration between MongoDB clusters, be applied equally to other kinds of data server cluster it
Between Data Migration.
Fig. 3 is referred to, it is the data migration method schematic diagram according to one embodiment of the invention, the data are and to move
The related target data of the business of shifting, carries out two data server clusters of Data Migration, and it can for example be retouched for above figure 1
The first data server cluster 200 and the second data server cluster 300 stated, or the first of the description of above figure 2
The MongoDB clusters 600 of MongoDB clusters 500 and the 2nd.
The data migration method starts from step S310, in step S310, by what is stored in the first data server cluster
The target data write before first time point and the first time point imports the second data server cluster.Specifically,
When Data Migration is ready for, selection one first has been the first time point of past tense, then by the first data, services
Target data is written of before the first time point and first time point that are stored in device cluster imported into the second data server
Cluster.For example, the target data in the first data server cluster can be backuped to first the storage medium of look-alike disk, such as
Data importing can be carried out in MongoDB clusters by this backup tool using mongodump；Then again by storage medium
Target data imports the second data server cluster, such as this is extensive can to use mongorestore in MongoDB clusters
Return to work tool.Optionally, if there is master data server in the first data server cluster and from data server, then preferably
Cutting off one from after the write operation of data server, then target data is derived from data server from this.This step
Can be performed by the primary data import modul 702 in the primary data import modul 102 or Fig. 2 in earlier figures 1, it is related
Technology is realized may be referred to foregoing primary data import modul associated description in embodiments, and here is omitted.
The number of targets of the second data server cluster is backed up to from the first data server cluster in previous step S310
According to being that the first data server cluster was written to before first time point and first time point, therefore follow-up need migration
Be exactly the target data for writing the first data device cluster after the first time.And then, in step s 320, first-selection obtain with
The associated Operation Log of the target data of the first data server cluster is write after the first time point, Operation Log is
The information record that each secondary data writes this operation particular content is recorded, including writing the first data server each time
The content of the target data of cluster, generally also includes the timestamp of write-in, if the operation is in the data that were previously written
The renewal of appearance, then the new value after updating, the also old value before record renewal are not only recorded in Operation Log.As can be seen here, root
According to Operation Log it is known that writing the particular content of data every time.During Operation Log is obtained, because data are
It is written of one by one, and then Operation Log is also to produce one by one, therefore can in real time obtain Operation Log, i.e., often write one
Or several Operation Logs, just go to obtain once；Operation Log can also periodically be obtained.Step S320 can be by earlier figures 1
Synchronization module 104 or Fig. 2 in synchronization module 704 perform, related technology is realized may be referred to aforementioned synchronization module each
Associated description in embodiment, here is omitted.
Then, in step S330, the first data, services are write after first time point according to what step S320 was obtained
The Operation Log of device cluster, updates the target data in the second data server cluster.Specifically, can be according to Operation Log
Corresponding data are also written to the second data server cluster by the particular content of each write-in data of middle record, so that
Also the target that the first data server cluster is write after first time point is successfully stored in second data server cluster
Data.Because the content of every Operation Log is less, therefore step S330 is finished and updates the speed of operation also just quickly, closely
Every write-in one target data of the first data server cluster can be reached, the second data server cluster is also written into therewith
In.So far, the target data in the first data server cluster, has all been migrated to the second data server cluster,
And reach the synchronous purpose of the target data in two data server clusters.Step S330 can be by earlier figures 1
Update module 106 or Fig. 2 in update module 706 perform, related technology is realized may be referred to foregoing update module each
Associated description in embodiment, here is omitted.
Hereafter, can also detect whether the target data stored in two data server clusters is consistent, and by behaviour
The auxiliary informations such as the numerical value content before and after making timestamp in daily record and updating, judge that two data server clusters are in the lump
No synchronized success.The step can be performed by the synchronous detection module described in migration equipment 100 above.If inspection
Synchronized success is determined after survey, then can will just connect the entry address of data server by the first data server cluster
Entry address be changed to the entry address of the second data server cluster.The step can be by migration equipment above
Address update module is performed.Hereafter, no matter application server is to need to write target data, or target data is entered
Row reads, and all can perform corresponding operation by the second data server cluster.In two data servers of observation a period of time
After operation complete stability, it is possible to delete the target data in the first data server cluster.
It should be noted that the order of each step in the above method is adjustable, such as step S320 can not
Step S310 must be waited just to start after having performed completely, in fact it could happen that the two steps are performed simultaneously or partial period is performed simultaneously
Situation.Because Operation Log is to produce in real time, therefore step S320 and step S330 can also be repeated, i.e., whenever one
After bar or several Operation Logs are newly produced, a step S320 and step S330 is carried out, to obtain newest operation in time
Target data in daily record and second data server cluster that upgraded in time according to Operation Log, such that it is able to take two data
Business device reaches being synchronously written for target data as early as possible.Again for example, previously mentioned synchronous detecting step, it is also possible to repeat,
No successful synchronization is found after being such as finished for the first time, then can also be detected again after certain interval of time, directly
To two data server cluster synchronizations are detected.
By the description of various embodiments above, using technical scheme provided in an embodiment of the present invention, in migrating data
During target data before the possible larger first time point of amount and first time point, it is not necessary to which stopping will migrate business
Data being normally written and reading, such as application server still can be to legacy data server cluster（As previously described
First data server cluster）In a data server carry out data write-in and reading, while from legacy data server set
Another data server in group derives target data before first time point and first time point to new data service device
Cluster（Second data server cluster as previously described）；Legacy data server cluster is write after first time point is migrated
Target data during, be by the target data synchronization after the very first time according to the Operation Log of legacy data server cluster
It is written in new data service device, in the process again without the business to be migrated is stopped, in two new and old data, services
After device cluster Complete Synchronization, the business to be migrated just directly can be normally written and read data to new data service device cluster
.As can be seen here, using technical scheme, in the transition process of whole business, need not all stop what is being migrated
Business, and then the external normal service of the business and operation would not be influenceed, it is achieved thereby that real by backup need not be shut down
The beneficial effect of existing Data Migration.
Algorithm and display be not inherently related to any certain computer, virtual system or miscellaneous equipment provided herein.
Various general-purpose systems can also be used together with based on teaching in this.As described above, construct required by this kind of system
Structure be obvious.Additionally, the present invention is not also directed to any certain programmed language.It is understood that, it is possible to use it is various
Programming language realizes the content of invention described herein, and the description done to language-specific above is to disclose this hair
Bright preferred forms.
In specification mentioned herein, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be put into practice in the case of without these details.In some instances, known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this description.
Similarly, it will be appreciated that in order to simplify one or more that the disclosure and helping understands in each inventive aspect, exist
Above to the description of exemplary embodiment of the invention in, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：I.e. required guarantor
The application claims of shield features more more than the feature being expressly recited in each claim.More precisely, such as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, and wherein each claim is in itself
All as separate embodiments of the invention.
Those skilled in the art are appreciated that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more equipment different from the embodiment.Can be the module or list in embodiment
Unit or component be combined into a module or unit or component, and can be divided into addition multiple submodule or subelement or
Sub-component.In addition at least some in such feature and/or process or unit exclude each other, can use any
Combination is to this specification（Including adjoint claim, summary and accompanying drawing）Disclosed in all features and so disclosed appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification（Including adjoint power
Profit requires, makes a summary and accompanying drawing）Disclosed in each feature can carry out generation by the alternative features for providing identical, equivalent or similar purpose
Replace.
Although additionally, it will be appreciated by those of skill in the art that some embodiments in this include institute in other embodiments
Including some features rather than further feature, but the combination of the feature of different embodiments means in the scope of the present invention
Within and form different embodiments.For example, in the following claims, embodiment required for protection it is any it
One mode can use in any combination.
All parts embodiment of the invention can be realized with hardware, or be run with one or more processor
Software module realize, or with combinations thereof realize.It will be understood by those of skill in the art that can use in practice
Microprocessor or digital signal processor（DSP）It is according to embodiments of the present invention between data server cluster to realize
Carry out some or all functions of some or all parts in the migration equipment of Data Migration.The present invention can also be realized
It is some or all equipment or device program for performing method as described herein（For example, computer program
And computer program product）.It is such to realize that program of the invention be stored on a computer-readable medium, or can have
There is the form of one or more signal.Such signal can be downloaded from internet website and obtained, or in carrier signal
Upper offer, or provided in any other form.
It should be noted that above-described embodiment the present invention will be described rather than limiting the invention, and ability
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol being located between bracket should not be configured to limitations on claims.Word "comprising" is not excluded the presence of not
Element listed in the claims or step.Word "a" or "an" before element is not excluded the presence of as multiple
Element.The present invention can come real by means of the hardware for including some different elements and by means of properly programmed computer
It is existing.In the unit claim for listing some equipment, several in these equipment can be by same hardware branch
To embody.The use of word first, second, and third does not indicate that any order.These words can be explained and run after fame
Claim.
Claims (12)
1. a kind of data mover system, for carrying out Data Migration between data server cluster, the system at least includes
First data server cluster and the second data server cluster, and migration equipment, wherein,
The data are the target data related to the business to be migrated；
The migration equipment includes：
Primary data import modul, be configured as will in the first data server cluster store first time point and this first
The target data write before time point imports the second data server cluster, described to be stored in first data server cluster
The target data write before first time point and the first time point import the second data server cluster and include institute
State in the first data server cluster the first first time point stored from data server and write before the first time point
The target data for entering imports second data server cluster, wherein, first data server cluster includes main number
According to server and some from data server；
Synchronization module, is configured as the mesh for obtaining with first data server cluster being write after the first time point
The associated Operation Log of mark data, the Operation Log at least includes application server after the first time point every time
Write the content of the target data in first data server cluster；And
Update module, the Operation Log for being configured as being obtained according to the synchronization module updates second data server cluster
In target data；
Synchronous detection module, is configured as based on the target data and second data in first data server cluster
Target data in server cluster, and/or, based on the write-in target data in the Operation Log being associated with target data
The forward and backward value of timestamp, updating target data, detects first data server cluster and the second data server collection
Whether the target data of group is synchronous；
Address update module, is configured as detecting first data server cluster and described in the synchronous detection module
After the synchronization of the target data of the second data server cluster, the entry address for connecting data server is counted by described first
The entry address of second data server cluster is changed to according to the entry address of server cluster.
2. data mover system according to claim 1, the primary data import modul includes：
First primary data imports submodule, when being configured as will be stored in first data server cluster described first
Between point and the first time point before write target data import a storage medium；
Second primary data imports submodule, is configured as the target data that is directed into storage medium importing described the
Two data server clusters.
3. data mover system according to claim 2, wherein, first primary data imports submodule and is configured to
Mongodump will write before the first time point stored in first data server cluster and the first time point
Target data back up to a storage medium；And
Second primary data imports the mesh that submodule is configured to during mongorestore will back up to the storage medium
Mark data import second data server cluster.
4., according to the data mover system of any one of claims 1 to 3, the migration equipment also includes：
Disable processing module, be configured as after first time point stopping in first data server cluster first from
The write operation of data server.
5., according to the data mover system of any one of claims 1 to 3, the Operation Log is also including in following information
Plant or various：
Write the timestamp of data；And
When the data of write-in are the renewals to legacy data, the old value before data renewal.
6., according to the data mover system of any one of claims 1 to 3, first data server cluster is first
MongoDB clusters, second data server cluster is the 2nd MongoDB clusters, and the Operation Log is in MongoDB
oplog。
7. a kind of data migration method, for carrying out Data Migration between data server cluster, the data are and to move
The related target data of the business of shifting, the method includes：
The target data write before first time point and the first time point that will be stored in first data server cluster
Import the second data server cluster, it is described will be stored in first data server cluster first time point and this first when
Between put before the target data that writes import the second data server cluster and include in first data server cluster the
The target data write before one first time point stored from data server and the first time point imports described second
Data server cluster, wherein, first data server cluster includes master data server and some from data, services
Device；
The operation day that acquisition is associated with the target data that first data server cluster is write after first time point
Will, the Operation Log at least writes first data, services every time including application server after the first time point
The content of the target data in device cluster；
Target data in second data server cluster is updated according to the Operation Log for obtaining；
Based on the number of targets in the target data in first data server cluster and second data server cluster
According to, and/or, timestamp, updating target data based on the write-in target data in the Operation Log being associated with target data
Whether the target data of forward and backward value, detection first data server cluster and second data server cluster
It is synchronous；And
After the target data synchronization of first data server cluster and second data server cluster, by even
The entry address for connecing data server is changed to the second data clothes by the entry address of first data server cluster
The entry address of business device cluster.
8. method according to claim 7, it is described will be stored in first data server cluster first time point and this
The step of target data write before one time point imports second data server cluster includes：
The number of targets that will be write before the first time point stored in first data server cluster and the first time point
According to importing a storage medium；And
The target data being directed into storage medium is imported into the second data server cluster.
9. method according to claim 8, wherein,
By mongodump by before the first time point stored in first data server cluster and the first time point
The target data of write-in backs up to a storage medium；And
The target data that will be backed up to by mongorestore in the storage medium imports the second data server collection
Group.
10., according to the method for any one of claim 7 to 9, methods described also includes：
In stopping first data server cluster after the first time point first from the write-in of data server
Operation.
11. according to the method for any one of claim 7 to 9, and the Operation Log also includes the one kind or many in following information
Kind：
Write the timestamp of data；And
When the data of write-in are the renewals to legacy data, the old value before data renewal.
12. according to the method for any one of claim 7 to 9, and first data server cluster is a MongoDB collection
Group, second data server cluster is the 2nd MongoDB clusters, and the Operation Log is the oplog in MongoDB.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210429724.9A CN102982085B (en) | 2012-10-31 | 2012-10-31 | Data mover system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210429724.9A CN102982085B (en) | 2012-10-31 | 2012-10-31 | Data mover system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102982085A CN102982085A (en) | 2013-03-20 |
| CN102982085B true CN102982085B (en) | 2017-05-31 |
Family
ID=47856105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210429724.9A Expired - Fee Related CN102982085B (en) | 2012-10-31 | 2012-10-31 | Data mover system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102982085B (en) |
Families Citing this family (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8533155B2 (en) * | 2009-10-30 | 2013-09-10 | Hitachi Data Systems Corporation | Fixed content storage within a partitioned content platform, with replication |
| US9747311B2 (en) | 2013-07-09 | 2017-08-29 | Oracle International Corporation | Solution to generate a scriptset for an automated database migration |
| US11157664B2 (en) | 2013-07-09 | 2021-10-26 | Oracle International Corporation | Database modeling and analysis |
| US9792321B2 (en) * | 2013-07-09 | 2017-10-17 | Oracle International Corporation | Online database migration |
| US10776244B2 (en) | 2013-07-09 | 2020-09-15 | Oracle International Corporation | Consolidation planning services for systems migration |
| US9996562B2 (en) | 2013-07-09 | 2018-06-12 | Oracle International Corporation | Automated database migration architecture |
| US9805070B2 (en) | 2013-07-09 | 2017-10-31 | Oracle International Corporation | Dynamic migration script management |
| CN103455589B (en) * | 2013-08-29 | 2016-09-21 | 中国农业银行股份有限公司 | Product data moving method under product factory mode, Apparatus and system |
| CN104486373A (en) * | 2014-11-21 | 2015-04-01 | 华为技术有限公司 | Lock resource migration method, nodes and distributed system |
| CN104462484B (en) * | 2014-12-18 | 2018-05-22 | 北京奇虎科技有限公司 | Data processing method, data processor and system |
| CN105824846B (en) * | 2015-01-09 | 2021-04-13 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN105989006A (en) * | 2015-01-27 | 2016-10-05 | 中芯国际集成电路制造(上海)有限公司 | Data migration method and device |
| CN104765794B (en) * | 2015-03-25 | 2018-10-09 | 深圳市创梦天地科技有限公司 | A kind of method for partitioning database and device |
| CN106682021B (en) * | 2015-11-09 | 2021-09-28 | 五八同城信息技术有限公司 | Database migration method and device |
| CN105472045A (en) * | 2016-01-26 | 2016-04-06 | 北京百度网讯科技有限公司 | Database migration method and database migration device |
| US10620834B2 (en) * | 2016-03-25 | 2020-04-14 | Netapp, Inc. | Managing storage space based on multiple dataset backup versions |
| US11036696B2 (en) | 2016-06-07 | 2021-06-15 | Oracle International Corporation | Resource allocation for database provisioning |
| CN106850821A (en) * | 2017-02-21 | 2017-06-13 | 山东浪潮云服务信息科技有限公司 | A kind of data managing method, data sending device, data sink and system |
| CN108664496B (en) * | 2017-03-29 | 2022-03-25 | 腾讯科技（深圳）有限公司 | Data migration method and device |
| CN107247727A (en) * | 2017-04-28 | 2017-10-13 | 北京五八信息技术有限公司 | Data migration method and equipment |
| CN107153699B (en) * | 2017-05-09 | 2020-10-02 | 北京恒华伟业科技股份有限公司 | Method and device for dynamically expanding cluster server |
| CN107220375B (en) * | 2017-06-20 | 2020-02-11 | 北京星选科技有限公司 | Data reading and writing method and server |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107391635A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data synchronous system and method |
| CN107391634B (en) * | 2017-06-30 | 2021-04-09 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107391628B (en) * | 2017-06-30 | 2021-04-09 | 北京奇虎科技有限公司 | Data synchronization method and device |
| CN107479948A (en) * | 2017-08-18 | 2017-12-15 | 郑州云海信息技术有限公司 | A kind of business migration method and device |
| CN109445718A (en) * | 2018-11-16 | 2019-03-08 | 广东小天才科技有限公司 | A kind of method for writing data and system based on Data Migration |
| US11256671B2 (en) | 2019-09-13 | 2022-02-22 | Oracle International Corporation | Integrated transition control center |
| CN115248746A (en) * | 2021-04-26 | 2022-10-28 | 华为云计算技术有限公司 | Data backup method, system and related equipment |
| CN113946293A (en) * | 2021-10-27 | 2022-01-18 | 北京达佳互联信息技术有限公司 | Cluster data migration method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081552A (en) * | 2009-12-01 | 2011-06-01 | 华为技术有限公司 | Method, device and system for transferring from physical machine to virtual machine on line |
| CN102917072A (en) * | 2012-10-31 | 2013-02-06 | 北京奇虎科技有限公司 | Device, system and method for carrying out data migration between data server clusters |
-
2012
- 2012-10-31 CN CN201210429724.9A patent/CN102982085B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102081552A (en) * | 2009-12-01 | 2011-06-01 | 华为技术有限公司 | Method, device and system for transferring from physical machine to virtual machine on line |
| CN102917072A (en) * | 2012-10-31 | 2013-02-06 | 北京奇虎科技有限公司 | Device, system and method for carrying out data migration between data server clusters |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102982085A (en) | 2013-03-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102982085B (en) | Data mover system and method | |
| CN102917072B (en) | For carrying out the equipment of Data Migration, system and method between data server cluster | |
| CN103530290B (en) | Data migration method and system between database | |
| US9183268B2 (en) | Partition level backup and restore of a massively parallel processing database | |
| CN102662946B (en) | The method and system of change is recorded in a kind of automatic monitoring data storehouse | |
| CN106844682B (en) | Method for interchanging data, apparatus and system | |
| CN105868343B (en) | Database migration method and system | |
| CN102541757B (en) | Write cache method, cache synchronization method and device | |
| CN105279285B (en) | A kind of synchronization system and method for relevant database and non-relational database | |
| CN105843702A (en) | Method and device for data backup | |
| CN104657459A (en) | Massive data storage method based on file granularity | |
| CN104281717B (en) | A kind of method for setting up magnanimity ID mapping relations | |
| CN106933703A (en) | A kind of method of database data backup, device and electronic equipment | |
| CN110321383A (en) | Big data platform method of data synchronization, device, computer equipment and storage medium | |
| CN104199963A (en) | Method and device for HBase data backup and recovery | |
| CN104572856A (en) | Converged storage method of service source data | |
| CN106484716A (en) | Organizational structure method of adjustment and system | |
| CN104484131B (en) | The data processing equipment of multiple disks server and corresponding processing method | |
| CN104866521A (en) | Apparatus For Processing Query In Database With Hybrid Storages | |
| CN102567204A (en) | Method for managing automatic testing data | |
| CN106155838B (en) | A kind of database back-up data restoration methods and device | |
| CN103365740B (en) | A kind of data cold standby method and device | |
| CN106780157B (en) | Ceph-based power grid multi-temporal model storage and management system and method | |
| CN106682021A (en) | Database migration method and device | |
| CN105955989A (en) | Method for establishing master and slave servers of cloud platform database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170531 Termination date: 20211031 |