CN104424283A - Data migration system and data migration method - Google Patents
Data migration system and data migration method Download PDFInfo
- Publication number
- CN104424283A CN104424283A CN201310388988.9A CN201310388988A CN104424283A CN 104424283 A CN104424283 A CN 104424283A CN 201310388988 A CN201310388988 A CN 201310388988A CN 104424283 A CN104424283 A CN 104424283A
- Authority
- CN
- China
- Prior art keywords
- migration
- data
- cluster
- units
- journal file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/563—Data redirection of data network streams
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration system and a data migration method which are applied for data migration between the main cluster and the standby cluster of a distributed database; the system comprises a main cluster, and one or more standby clusters, wherein the main cluster comprises one or more servers, each server has an area management unit, each area management unit generates at least one log file which is correlated to the data to be migrated, and wherein the one or more servers of the main cluster or the standby clusters are provided with one or more migration units; the one or more migration units are used for writing the data to be migrated into a designated standby cluster through parsing the log file which is correlated to the data to be migrated. By employing the technical scheme, the performance pressure of the main cluster can be reduced, not only the continuous data migration can be supported, but also the data migration in designated period can be supported, thus solving the data reinforce in the designated period.
Description
Technical field
The application relates to distributed data base field, particularly relates to a kind of system of Data Migration and the method for Data Migration.
Background technology
HBase is distributed data base, generally comprise standby (Slave) cluster that master's (Master) cluster of read-write service and main cluster are externally provided, when main cluster goes wrong, general needs are switched to standby cluster, externally provide read-write service by standby cluster, main cluster needs data copy (copying) in standby cluster.Suppose there are two clusters: M cluster and S cluster, wherein, M is main cluster, and S is standby cluster.Open main cluster M copying (replication) to standby cluster S.When M cluster cannot provide service for a certain reason, all read-write requests can be switched on S cluster.Consider two scenes: one, some data in M cluster also do not have enough time to copy in S cluster to go, and after being so switched to S cluster, this part data is inconsistent, also just result in the unavailability of this part data.Its two, after M cluster recovery is normal, when all read-write requests are switched back M cluster, have partial data exist in S cluster and do not exist in M cluster, this part data is inconsistent, also can cause the unavailability of this part data.Further, in above two scenes, also need to consider: (replication) function that copies of existing scheme is deployed in M cluster, and M cluster is the main cluster externally providing read-write service, this can affect the readwrite performance of main cluster.In addition, how to realize the real-time reconciliation of data, that is, in the synchronizing process of data, how to ensure that data syn-chronization has been gone over, do you how to confirm that whether the data in active and standby cluster consistent? between existing cluster in Data Migration, can not confirm the whether consistent scheme of Data Data between active and standby cluster.
Also there is no a kind of problem that can solve generation when above scene occurs so far, to improve the scheme of cluster availability.
Such as, existing HBase system, generally the data in main cluster are write asynchronously standby cluster by resolving journal file (Hlog), the machine of each main cluster there is a district management unit (such as, RegionServer, is called for short RS), be responsible for the read-write requests of process user, the corresponding journal file (Hlog) of each district management unit, Hlog leaves in Hadoop distributed file system (HDFS).Under newly-generated Hlog can be joined the Zookeeper path of respective regions administrative unit by HBase, the Hlog that simultaneously each district management unit generates has one to copy (replication) thread to resolve, and the data parsed write in the Slave cluster of specifying, and record skew (offset) position of treated Hlog.After processing certain Hlog, corresponding Hlog node can be deleted on Zookeeper.After certain district management unit delays machine, the district management unit having other then processes corresponding Hlog, the chance of the Hlog of the district management unit of this machine of delaying of process is mainly obtained by fighting for lock (lock) node, then under the Hlog under the district management unit path of the machine of delaying being copied to the path of oneself, and by the route deletion of the district management unit of the machine of delaying.
Existing HBase cluster, data Replica (replication) in main cluster being had following shortcoming to the process of standby cluster: 1, only support initiatively to copy data to from main cluster Push() data are to for cluster (push-model), this just determines that the process need copied carries out on main cluster, therefore, the performance of main cluster can be affected.2, this pattern of Data Migration continued only is supported, do not support the Data Migration pattern of fixed time section, therefore, the polishing of the data of fixed time section cannot be met, such as, when main cluster cannot provide service for a certain reason, when there is the situations such as machine room power-off, if the partial data of main cluster does not also have enough time to write standby cluster, the partial data of standby cluster can be caused unavailable.3, district management unit delays machine by Zookeeper(distributed management coherent system) event notice, the lock of the district management unit of the machine of delaying all is fought for after all district management unit obtain event notice, not automatic equalization mechanism in this process, may cause a district management cell processing much to delay the Hlog of district management unit of machine.In the process of 4, Data Migration, cannot confirm whether data have synchronously pass by, cannot confirm that whether two company-data are consistent, namely not realize reconciliation function.
For current by the process of the data Replica of main cluster to standby cluster, need to ensure the availability of data in active-standby switch process and consistance and support the scheme of polishing of data of fixed time section.
Summary of the invention
The fundamental purpose of the application is to provide a kind of system of Data Migration and the method for Data Migration, and the main cluster existed to solve prior art is switched to availability and the consistency problem of data in standby colonization process, wherein:
The system of a kind of Data Migration that the application provides, for carrying out Data Migration between the main cluster and standby cluster of distributed data base, this system comprises: main cluster and one or more standby cluster; Wherein, described main cluster comprises one or more server, and each server has district management unit, and each district management unit generates at least one and the journal file wanting migration data relevant; Described journal file comprises: describedly want the content of migration data, described migration data of wanting writes the temporal information of described main cluster; And the one or more servers wherein, in described main cluster or standby cluster are configured with one or more migration units; Described one or more migration units, for by resolving the journal file of wanting migration data relevant to described thus by the described standby cluster wanting migration data to write to specify.
According to the embodiment of the application, within the system, described one or more migration units and described one or more district management unit are configured on same station server correspondingly.
According to the embodiment of the application, within the system, described one or more migration units, wherein any one is main migration units, distributes to described one or more migration units process for what generated by described one or more district management unit to the described journal file wanting migration data relevant.
According to the embodiment of the application, within the system, whether each migration units in described one or more migration units comprises: monitoring module, normal for the running status monitoring each migration units in described one or more migration units when this migration units becomes main migration units.
According to the embodiment of the application, within the system, described migration units comprises: acquisition module, that generate to the described journal file wanting migration data relevant for obtaining corresponding district management unit under the Data Migration pattern continued; Parsing module, for resolving described journal file, wants migration data described in obtaining; And writing module, for described migration data of wanting is write in the standby cluster of specifying.
According to the embodiment of the application, within the system, described acquisition module, it is that require to the described journal file wanting migration data relevant that the temporal information also generated for obtaining corresponding district management unit under the Data Migration pattern of at the appointed time section meets fixed time section; Described parsing module, also meets for resolving described temporal information the journal file that described fixed time section requires, wants migration data with what obtain described fixed time section; And said write module, also for the migration data of wanting in described fixed time section is write in the standby cluster of specifying.
According to the embodiment of the application, within the system, said write module, also for resolving in the increment list of the incremental data write correspondence that described journal file obtains.
According to the embodiment of the application, within the system, described migration units also comprises: mode conversion module, for the mode of operation of described migration units being changed between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.
According to the embodiment of the application, within the system, described mode conversion module comprises: transform subblock, for suspending described lasting Data Migration pattern, and opens the Data Migration pattern of described fixed time section.
According to the embodiment of the application, within the system, described mode conversion module also comprises: preserve submodule, for preserving the positional information of the part correspondence in described journal file be not processed in the journal file that has not been processed and described journal file when suspending described lasting Data Migration pattern; And record sub module, for recording the minimum time point when suspending described lasting Data Migration pattern in the journal file that finally generates of corresponding district management unit; Described transform subblock, be further used for the Data Migration pattern suspending described fixed time section, and open described lasting Data Migration pattern according to the positional information corresponding in described journal file of the part be not processed in the described journal file be not processed and described minimum time point.
According to the embodiment of the application, within the system, described migration units comprises: consistance contrast module, for contrasting data in described main cluster and described standby cluster of specifying, to determine the consistance of active and standby company-data.
According to the embodiment of the application, within the system, described migration units comprises: dynamic configuration module, carries out dynamic-configuration for carrying out the migration table of Data Migration, increment list, migration units cluster switch, log path, many standby clustering functionalitys and consistance contrast switch to described appointment for cluster to described main cluster.
Provide a kind of method of Data Migration according to the another aspect of the application, for carrying out Data Migration between main cluster and standby cluster, the method comprises: one or more migration units is resolved and the journal file wanting migration data relevant; Described migration data of wanting writes in the standby cluster of specifying by described one or more migration units; Wherein, described main cluster comprises one or more server, and each server has district management unit, and each district management unit generates at least one and the described journal file wanting migration data relevant; Described journal file comprises: describedly want the content of migration data, described migration data of wanting writes the temporal information of described main cluster; Described one or more migration units is configured on the one or more servers in described main cluster or standby cluster.
According to the embodiment of the application, in the method, one or more district management unit of described one or more migration units and described main cluster are configured on same station server correspondingly.
According to the embodiment of the application, in the method, main migration units in described one or more migration units, what generated by described one or more district management unit distributes to described one or more migration units to the described journal file wanting migration data relevant and processes, wherein, described main migration units is any one in described one or more migration units.
According to the embodiment of the application, in the method, also comprise: whether the state that in the described one or more migration units of main migration units monitoring in described one or more migration units, each migration units is run is normal.
According to the embodiment of the application, in the method, described one or more migration units is resolved and is comprised to the step of the described journal file wanting migration data relevant: under lasting Data Migration pattern, obtains corresponding district management unit that generate to the described journal file wanting migration data relevant; Resolve described journal file, described in obtaining, want migration data; The described step wanting migration data to write in the standby cluster of specifying comprises by described one or more migration units further: write in the standby cluster of specifying by described migration data of wanting under the Data Migration pattern continued.
According to the embodiment of the application, in the method, described one or more migration units is resolved and is comprised further to the step of the described journal file wanting migration data relevant: under the Data Migration pattern of at the appointed time section, and obtaining temporal information that corresponding district management unit generates, to meet fixed time section requirement to the described journal file wanting migration data relevant; Resolve described temporal information and meet the journal file that described fixed time section requires, want migration data with what obtain described fixed time section; And the described step wanting migration data to write in the standby cluster of specifying comprises by described one or more migration units further: under the Data Migration pattern of at the appointed time section, the migration data of wanting in described fixed time section is write in the standby cluster of specifying.
According to the embodiment of the application, in the method, also comprise: described one or more migration units will be resolved in the increment list of the incremental data write correspondence that described journal file obtains.
According to the embodiment of the application, in the method, also comprise: mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.
According to the embodiment of the application, in the method, mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section, comprise: suspend described lasting Data Migration pattern, and open the Data Migration pattern of described fixed time section.
According to the embodiment of the application, in the method, mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section, also comprise: the positional information of preserving the part correspondence in described journal file be not processed in the journal file and described journal file be not processed when suspending described lasting Data Migration pattern, and the minimum time point in the journal file of record last generation that corresponding district management unit generates when suspending described lasting Data Migration pattern, suspend the Data Migration pattern of described fixed time section, and open described lasting Data Migration pattern according to the positional information corresponding in described journal file of the part be not processed in the described journal file be not processed and described minimum time point.
According to the embodiment of the application, in the method, also comprise: described one or more migration units contrasts data in described main cluster and described standby cluster of specifying, to determine the consistance of active and standby company-data.
According to the embodiment of the application, in the method, also comprise: described one or more migration units is carried out the migration table of Data Migration, increment list, migration units cluster switch, log path, many standby clustering functionality and consistance contrast switch to described appointment for cluster to described main cluster and carried out dynamic-configuration.
Compared with prior art, according to the technical scheme of the application, both can support the Data Migration of Push mode (push-model), also can support the Data Migration of Pull mode (pull-mode), the performance pressures of main cluster can be reduced; Not only can support the Data Migration continued, also can support the Data Migration of fixed time section, solve the Data-parallel language of fixed time section; Further, whether all right active and standby company-data of real time contrast is consistent.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide further understanding of the present application, and form a application's part, the schematic description and description of the application, for explaining the application, does not form the improper restriction to the application.In the accompanying drawings:
Fig. 1 is the schematic diagram on the migration units of the system of a kind of Data Migration of the embodiment of the present application is configured in main cluster one or more server;
Fig. 2 is that the migration units of the system of a kind of Data Migration of the embodiment of the present application is configured in for the schematic diagram on one or more server in cluster;
Fig. 3 is the structured flowchart of the migration units of the embodiment of the present application;
Fig. 4 is the process flow diagram of the method for the Data Migration of the embodiment of the present application;
Fig. 5 is the process flow diagram of a kind of embodiment of step S401 in Fig. 4 of the embodiment of the present application; And
Fig. 6 is the process flow diagram of the another kind of embodiment of step S401 in Fig. 4 of the embodiment of the present application.
Embodiment
The main thought of the application is, between main cluster and standby cluster, Data Migration is carried out by data migration unit, migration units can be deployed in main cluster, also standby cluster can be deployed in, be deployed in the performance pressures that standby cluster can reduce main cluster, meanwhile, the consistance of data in the active and standby cluster of real time contrast is consistent to ensure two company-data in data migration process.
For making the object of the application, technical scheme and advantage clearly, below in conjunction with the application's specific embodiment and corresponding accompanying drawing, technical scheme is clearly and completely described.Obviously, described embodiment is only some embodiments of the present application, instead of whole embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not making the every other embodiment obtained under creative work prerequisite, all belong to the scope of the application's protection.
According to the embodiment of the application, provide a kind of system of Data Migration.
The data mover system of the application may be used for carrying out Data Migration between the main cluster of distributed data base and standby cluster, especially can be applied to HBase(Hadoop Database) Data Migration between the main cluster of database and standby cluster.
According to the embodiment of the application, this data mover system can comprise: main cluster and one or more standby cluster, wherein, the one or more servers in main cluster or standby cluster can configure one or more migration units.
Main cluster can comprise one or more server, each server generally can have a district management unit, wherein each district management unit can generate at least one and the journal file wanting migration data relevant, and this journal file can be stored in distributed file system HDFS; Journal file comprises: the content of migration data, want migration data to write the temporal information (or the temporal information be modified) of described main cluster; When data are modified (such as, when wanting migration data because being modified by carrying out write operation), the time etc. of the data after change and change can write in journal file by district management unit, and described district management unit may be used for process user to the read-write requests of data in main cluster.
Described one or more migration units may be used for by resolve to obtain to the described journal file wanting migration data relevant described in want migration data, and described migration data of wanting to be write in the standby cluster of specifying.Wherein, the standby cluster of specifying can be one also can be multiple, when the standby cluster of specifying is one, the migration data of wanting obtained of being resolved by daily record is all write this in cluster.When the standby cluster of specifying is multiple, all write in each standby cluster of specifying respectively by being resolved all migration datas of wanting obtained by daily record, namely each standby cluster of specifying be written into by daily record resolve obtain all want migration data.
One or more servers in main cluster or standby cluster can configure described one or more migration units, that is, described migration units can be configured in described main cluster and also can be configured in described standby cluster, also can be configured on incoherent cluster.Should note, the standby cluster mentioned in the application can comprise any cluster except described main cluster, standby cluster can be with the standby cluster of described main cluster, also can be described main cluster standby cluster outside other clusters (other clusters can be any cluster incoherent with described main cluster).
According to the embodiment of the application, described one or more migration units, wherein any one is main migration units.Such as, in HBase database, when certain migration units obtains corresponding lock, then can become main migration units.Described main migration units, what may be used for described one or more district management unit to generate distributes to described one or more migration units to the described journal file wanting migration data relevant and processes.Specifically, described one or more migration units can form migration units cluster, described one or more district management unit can be distributed to corresponding migration units and process by main migration units, that is, make the corresponding one or more district management unit of described one or more migration units process generate to the journal file wanting migration data relevant.
According to the embodiment of the application, when described migration units is configured in one or more servers of main cluster, namely, when described migration units is configured in main cluster, the Data Migration of Push mode can be realized, that is, data are pushed to the standby cluster of specifying from main cluster.Alternatively, one or more district management unit of one or more migration units and main cluster can be configured on same station server accordingly.Can with reference to shown in figure 1, Fig. 1 is the schematic diagram that one or more district management unit of one or more migration units and main cluster are configured on same station server by the system of a kind of Data Migration of the embodiment of the present application accordingly.
As shown in Figure 1, the main cluster 10 of data mover system 100 has server 11-1 ~ 11-N, corresponding with one or more district management unit 13-1 ~ 13-N of main cluster 10 for one or more migration units 12-1 ~ 12-N is configured on same station server (physical machine) 11, a corresponding configuration migration units 12 on the server 11 at namely each district management unit 13 place, each migration units 12 is by reading and resolving the journal file of corresponding district management unit 13, will write in the standby cluster 20-1 specified in multiple standby cluster (standby cluster 20-1 ~ 20-M) by migration data.Wherein, the district management unit 13 of the corresponding process of each migration units 12 can be the district management unit with it on same station server 11, also can be the district management unit in an other physical machine, can be that each migration units 12(wherein can comprise this main migration units by main migration units) assign the district management unit 13 of its correspondence, when wherein certain migration units cannot normally work (such as, to delay machine) time, main migration units can also assign other migration units to process the corresponding district management unit of this migration units.Wherein, the standby cluster of specifying can be one also for multiple, can Figure 1 shows that situation when the standby cluster of specifying is, as shown in Figure 1, when the standby cluster 20 of specifying is one, the migration data of wanting obtained of being resolved by daily record is write in the standby cluster 20 of specifying.In addition, when the standby cluster of specifying is multiple, each migration units all writes in each standby cluster of specifying by being resolved the migration data of wanting obtained by daily record respectively, namely each standby cluster of specifying be written into by daily record resolve obtain all want migration data.It should be noted that, Figure 1 shows that the Alternate embodiments of the application, the embodiment of the application is not limited thereto, such as, when there being N number of district management unit in main cluster, can configure M migration units to process N number of district management unit on main cluster, N can be not equal to M.
According to the embodiment of the application, when described migration units is configured in one or more servers of standby cluster, namely, when migration units is configured in described standby cluster, the Data Migration of Pull mode can be realized, that is, from standby cluster by the data pull of main cluster to the standby cluster of specifying.Should note, the standby cluster mentioned in the application can comprise any cluster except described main cluster, standby cluster can be described main cluster standby cluster, also can be described main cluster standby cluster outside other clusters (that is, any cluster incoherent with described main cluster).Such as, one or more migration units can be configured in one or more servers of the standby cluster of specifying of main cluster or one or more migration units can be configured in one or more servers of other clusters outside described standby cluster of specifying.Can with reference to shown in figure 2, Fig. 2 is the schematic diagram that one or more migration units is configured in other clusters outside standby cluster that main cluster specifies by the system of a kind of Data Migration of the embodiment of the present application.As shown in Figure 2, the main cluster 50 of data mover system 200 has server 51-1 ~ 51-N, specifying at main cluster 50 in multiple server 31-1 ~ 31-N of the standby cluster 30 outside standby cluster 40 and arrange multiple migration units 32-1 ~ 32-N, is that each migration units 32(wherein can comprise this main migration units by main migration units) assign the district management unit 52 of its correspondence.Multiple migration units 32 is by reading and resolving the journal file wanting migration data to be correlated with of district management unit 52 generation corresponding in main cluster 50 thus will write in this standby cluster 40 of specifying by migration data.It should be noted that, although the migration units 32 be configured in standby cluster 30 shown in Fig. 2 is that quantity is equal with district management unit 52 in main cluster 50, and can be one to one, but the embodiment of the application is not limited thereto, the quantity of migration units and district management unit can be unequal, can be the district management unit that each migration units assigns (wherein can comprise this main migration units) its correspondence by main migration units.
As shown in Figure 3, Fig. 3 is the structured flowchart of the migration units according to the embodiment of the present application.According to an embodiment of the application, each migration units 300 in described one or more migration units can comprise acquisition module 310, parsing module 320 and writing module 330.
The data mover system of the application can carry out the Data Migration between main cluster and standby cluster under the Data Migration pattern continued, and namely realizes the Data Migration pattern continued.
Under the Data Migration pattern continued, described acquisition module 310, may be used under the Data Migration pattern continued, obtain corresponding district management unit that generate to the described journal file wanting migration data relevant.
Described parsing module 320, may be used for resolving described journal file, described in obtaining, wants migration data.
Said write module 330, may be used for described migration data of wanting to write in the standby cluster of specifying.
When migration units is configured in one or more servers of main cluster, when being namely configured in main cluster, that is, carry out Data Migration in Push mode.Such as, the situation of a corresponding configuration migration units on the server at each district management unit place, each migration units 300 reads the journal file of local district management unit by read module 310, and resolved by parsing module 320 pairs of journal files, parsing module 320 resolved by writing module 330 migration data of wanting that described journal file obtains again and write the standby cluster of specifying, wherein, the standby cluster of specifying can be specify one standby cluster, also can for the multiple standby cluster of specifying.
The Data Migration mode of this Push, can obtain migration data and also write in the standby cluster of specifying by reading journal file that the district management unit of migration units this locality generate and carrying out resolving, can reduce the reading pressure of network.
When migration units is configured in one or more servers of standby cluster, when being namely configured in standby cluster, the i.e. Data Migration of Pull mode, such as, one or more migration units can be configured in one or more servers of specifying standby cluster, or one or more migration units is configured in one or more servers of other clusters, other clusters can be any cluster incoherent with described main cluster.
Each migration units 300 in standby cluster read that the district management unit of long-range main cluster generates by read module 310 to the journal file wanting migration data relevant, and by parsing module 320, described journal file is resolved, parsing module 320 being resolved by writing module 330 migration data of wanting that described journal file obtains again writes in the standby cluster of specifying, wherein, the standby cluster of specifying can be specify one standby cluster, also can for the multiple standby cluster of specifying.
The Data Migration mode of this Pull, the journal file generated by the corresponding district management unit reading long-range main cluster also writes standby cluster according to certain rule after resolving, owing to being carry out the operations such as the parsing of data in standby cluster (other outside the standby cluster of specifying or the standby cluster of specifying are for cluster), can reduce to resolve and process journal file to marry again in the less standby cluster of pressure the impact that main cluster is read and write, reduce the performance pressures of main cluster.
In the above-mentioned Data Migration pattern continued, it is no matter the Data Migration of Push mode or Pull mode, can the journal file of migration data be wanted to carry out resolving and obtain the data that will move to write in the standby cluster of specifying described in constantly generating, the effect of closely real-time Data Migration can be reached.
The data mover system of the application except carrying out the Data Migration between main cluster and standby cluster under the above-mentioned Data Migration pattern continued, the Data Migration between main cluster and standby cluster can also be carried out under the Data Migration pattern of at the appointed time section, namely require to carry out Data Migration between main cluster and standby cluster according to described fixed time section.The Data Migration between main cluster and standby cluster is carried out under how the data mover system introducing the application below realizes the Data Migration pattern of at the appointed time section.
According to the embodiment of the application, under the Data Migration pattern of described fixed time section, described acquisition module 310, under can also being used for the Data Migration pattern of at the appointed time section, obtaining temporal information that corresponding district management unit generates, to meet fixed time section that require to the described journal file wanting migration data relevant.
Described parsing module 320, can also be used for resolving the journal file that described temporal information meets described fixed time section, wants migration data with what obtain described fixed time section.
Said write module 330, can also be used for the migration data of wanting in described fixed time section to write in the standby cluster of specifying.
Due to journal file comprise described want migration data to write main cluster temporal information or the temporal information that changes of data, therefore according to described temporal information, Data Migration can be carried out to the migration data of wanting in fixed time section.The Data Migration of this fixed time section can also be used for Data-parallel language when main cluster switches to standby cluster.Further, both can be used for the situation of above-mentioned migration units in main cluster and also can be used for the situation of migration units in standby cluster.Namely, under the Data Migration pattern of at the appointed time section, the read module 310 of migration units 300 is according to the fixed time section requirement of Data Migration, the temporal information that the district management unit reading Local or Remote generates meets the journal file that described fixed time section requires, parsing module 320 resolves described journal file, migration data is wanted in this fixed time section to obtain, parsing module 320 resolved by writing module 330 migration data of wanting that journal file obtains again to write in the standby cluster of specifying, which achieves the Data Migration of fixed time section.
According to the embodiment of the application, each migration units in described one or more migration units can also comprise: consistance contrast module, this module may be used for contrasting data in described main cluster and described standby cluster of specifying, to determine the consistance of active and standby company-data.
The data of migration data and described standby cluster of specifying of wanting of resolving the acquisition of described journal file can be made comparisons by described consistance contrast module, the data in active and standby cluster can also be compared.Such as, in HBase cluster, (, only have and insert and delete two kinds of basic models in HBase when being in the operation of intercalation model), the data of resolving the acquisition of described journal file are made comparisons with the data that standby cluster is inquired about, due in HBase, most operation is all intercalation model, does not need like this to inquire about main cluster, alleviates the performance pressures of main cluster, reach again the object of reconciliation (that is, the contrast of the data consistency of active and standby cluster).When being in the operation of puncturing pattern, then the data of active and standby cluster are made comparisons simultaneously.Consistance contrast module can be dynamically open.
According to the embodiment of the application, each migration units in described one or more migration units can also comprise dynamic configuration module, and this module may be used for dynamic-configuration migration table, increment list, migration units cluster switch, log path (Hlog Path), many standby clustering functionalitys and consistance contrast switch.Wherein, dynamic-configuration migration table can for recording the table wanting migration data, and increment list can be the table of recording increment data; Migration units cluster switch, according to the needs of actual conditions, can open and close migration units cluster; Log path (Hlog Path) is the store path of the journal file wanting migration data corresponding; Many standby clustering functionalitys, that is, increase one or more standby cluster; Consistance contrast switch, can for carrying out the switch of the opening and closing of consistance contrast.
According to the embodiment of the application, each migration units in described one or more migration units can also comprise monitoring module, this module may be used for when this migration units becomes main migration units, whether normally monitors the state that in described one or more migration units, each migration units is run.Can whether normal: the number of migration units if monitoring the running status of each migration units by collecting following index, just in the number of the migration units of migration data, which district management unit is each migration units processing, the number of district management unit, also untreated journal file Hlog sum, the number of each district management unit also untreated journal file Hlog.
According to the embodiment of the application, no matter be under duration data migration model or in fixed time segment data migration model, writing module can be resolved in increment list corresponding to incremental data write that described journal file obtains.Such as, in HBase database, by resolving Hlog, the data of respective table can be obtained (in HBase database, generally store data in the form of a table, by major key rowkey, be good for retrieve data also referred to as row), thus want migration data to write the timestamp (timestamp) of main cluster described in obtaining, the timestamp (timestamp) that can increase the main cluster of write before the former major key rowkey of data forms new major key rowkey, therefore, these newly-generated major keys and corresponding data can be write in the new table generated according to predetermined naming rule by writing module 330, namely in increment list, increment list can be used for inquiring about the former table of arbitrary period and be written with which data, i.e. incremental data.
The application had both supported the pattern of the Data Migration continued, and also supported the pattern of the Data Migration of fixed time section, can also realize the conversion of the Data Migration pattern of Data Migration pattern and the fixed time section continued.
According to the embodiment of the application, each migration units in described one or more migration units can also comprise mode conversion module, and this module may be used for the mode of operation of described migration units to change between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.
Described mode conversion module comprises: transform subblock, and this module may be used for suspending described lasting Data Migration pattern, and opens the Data Migration pattern of described fixed time section.When being transformed into fixed time segment data migration model from described duration data migration model, suspending described lasting Data Migration pattern, and opening the Data Migration pattern of described fixed time section.
According to the embodiment of the application, described mode conversion module can also comprise: preserve submodule and record sub module.
Preserving submodule, may be used for preserving the positional information of the part correspondence in described journal file be not processed in the journal file that has not been processed and described journal file when suspending described lasting Data Migration pattern.
Record sub module, may be used for the minimum time point in the record journal file that corresponding district management unit finally generates when suspending described lasting Data Migration pattern.
Further, described transform subblock, is further used for the Data Migration pattern suspending described fixed time section, and recovers the work of described lasting Data Migration pattern according to described minimum time point.
Specifically, when from duration data migration model to the Data Migration patten transformation of fixed time section, preserve the positional information of the part correspondence in described journal file be not processed in the journal file and described journal file do not processed when submodule preserves the Data Migration pattern suspended and continue, and the minimum time point recorded in the journal file that corresponding district management unit finally generates, when migration units by mode of operation from the Data Migration pattern recovery of fixed time section to the Data Migration pattern continued time, transform subblock suspends the Data Migration pattern of described fixed time section, and according to the positional information of the part be not processed in the described journal file be not processed correspondence in described journal file and the work of the described lasting Data Migration pattern of described minimum time point recovery, namely, continue the part be not processed of the journal file do not processed described in process, and the Data Migration restarting to continue according to described minimum time point (also has newly-generated with the described journal file wanting migration data corresponding from described minimum time point, Data Migration is proceeded) from this time point.
Present invention also provides a kind of method of Data Migration.
Fig. 4 schematically shows the process flow diagram of the method for the Data Migration according to the application's embodiment.
According to the embodiment of the application, the method may be used for carrying out Data Migration between the main cluster and standby cluster of distributed data base, especially HBase(Hadoop Database can be applied to) Data Migration between the main cluster of database and standby cluster, as shown in the figure:
In step S401, one or more migration units is resolved and the described journal file wanting migration data relevant, and wherein, described one or more migration units is configured on the one or more servers in described main cluster or standby cluster.
In step S402, described migration data of wanting writes in the standby cluster of specifying by described one or more migration units.
Wherein, described main cluster can comprise one or more server, and each server generally can have a district management unit, and each district management unit can generate at least one and the described journal file wanting migration data relevant; Described journal file can comprise: describedly want the content of migration data, described migration data of wanting writes the temporal information (or the temporal information be modified) of described main cluster; When data are modified (such as, when wanting migration data to be modified because carrying out write operation or other operations), district management unit can by this journal file of the write such as time of the data after change and change.
Migration units can be configured in one or more servers of described main cluster, also can be configured in one or more servers of described standby cluster.Should note, the standby cluster mentioned in the application can comprise any cluster except described main cluster, standby cluster can be described main cluster standby cluster, also can be described main cluster standby cluster outside other clusters (that is, any cluster incoherent with described main cluster).When migration units is configured in one or more servers of described main cluster, that is, when migration units is configured in main cluster, the Data Migration of Push mode can be realized, push to the standby cluster of specifying from main cluster by data.Alternatively, can corresponding with one or more district management unit of main cluster for one or more migration units be configured on same station server, that is, the server at each district management unit place of main cluster configure a migration units.When migration units is configured in one or more servers of standby cluster, when namely migration units is configured in standby cluster, the Data Migration of Pull mode can be realized, namely from standby cluster by the data pull of main cluster to standby cluster.Such as, multiple migration units can be configured in one or more servers of the standby cluster of specifying of main cluster or described migration units is configured in outside standby cluster that described main cluster specifies other for cluster (described other for cluster, may also be referred to as other clusters, refer to any cluster outside the described main cluster wanting the original place of migration data) one or more servers in.Wherein, the standby cluster of specifying can be one also can be multiple, when the standby cluster of specifying is one, the migration data of wanting obtained of being resolved by daily record is all write this in cluster.When the standby cluster of specifying is multiple, all write in each standby cluster of specifying respectively by being resolved all migration datas of wanting obtained by daily record, namely each standby cluster of specifying be written into by daily record resolve obtain all want migration data.
According to the embodiment of the application, what described one or more district management unit generated by described main migration units distribute to described one or more migration units to the journal file wanting migration data relevant processes.Specifically, described one or more migration units can form a cluster, wherein any one migration units is main migration units, other migration units is from migration units, any migration units all may become main migration units, such as, in HBase database, when certain migration units obtains corresponding lock, main migration units will be become.Described one or more district management unit can be distributed to corresponding migration units and process by main migration units, that is, make one or more district management unit described in described one or more migration units alignment processing generate to the journal file wanting migration data relevant.
According to the embodiment of the application, the application can carry out the Data Migration between main cluster and standby cluster under the Data Migration pattern continued, between main cluster and standby cluster, namely carry out the Data Migration continued.
Under the Data Migration pattern continued, step S401 can comprise step S501-S502.With reference to the step S401 process flow diagram that figure 5, Fig. 5 is according to the application one embodiment.
As shown in Figure 5, in step S501, under the Data Migration pattern continued, obtain corresponding district management unit that generate to the described journal file wanting migration data relevant.That is, under the Data Migration pattern continued, the journal file that the district management unit reading the correspondence that main migration units is distributed generates.
In step S502, by resolving described journal file, described in obtaining, want migration data.That is, migration data is wanted described in the mode of resolving based on daily record obtains.
Further, in step S402, under described lasting Data Migration pattern, described migration data of wanting is write in the standby cluster of specifying.
Above-mentioned lasting migration model, can for constantly generate to the described journal file wanting migration data relevant carry out resolving obtain described in want migration data to carry out the migration continued, closely real-time Data Migration can be reached.And, namely can be used for migration units situation about being arranged in main cluster also to can be used for migration units and be arranged on situation in standby cluster, that is this lasting Data Migration pattern both may be used for the Data Migration of Push mode, also may be used for the Data Migration of Pull mode.
According to another embodiment of the application, the application can also carry out the Data Migration between main cluster and standby cluster under the Data Migration pattern of at the appointed time section, between main cluster and standby cluster, namely carry out the migration of the data in fixed time section.
Under the Data Migration pattern of at the appointed time section, step S401 can comprise step S601-S602.With reference to figure 6, it is the step S401 process flow diagram according to another embodiment of the application.
As shown in Figure 6, in step s 601, under the Data Migration pattern of at the appointed time section, obtaining temporal information that corresponding district management unit generates, to meet fixed time section that require to the described journal file wanting migration data relevant.
Because journal file comprises the temporal information wanting the temporal information of the described main cluster of migration data write or data to change, therefore according to described temporal information, the journal file met in fixed time section can be obtained.
In step S602, resolve described temporal information and meet the journal file that described fixed time section requires, want migration data with what obtain described fixed time section.
Further, in step S402, under the Data Migration pattern of described fixed time section, the migration data of wanting in described fixed time section is write in the standby cluster of specifying.
Above-mentioned Data Migration mode is that the journal file by resolving fixed time section obtains the Data Migration wanting migration data to carry out, and the Data Migration of this fixed time section can also be used for Data-parallel language when main cluster switches to standby cluster.And, namely can be used for migration units situation about being arranged in main cluster also to can be used for migration units and be arranged on situation in standby cluster, that is the Data Migration of this fixed time section both may be used for the Data Migration of Push mode, also may be used for the Data Migration of Pull mode.
According to the embodiment of the application, no matter be under the Data Migration pattern continued or under the Data Migration pattern of fixed time section, the step that described one or more migration units will be resolved in the increment list of the incremental data write correspondence that described journal file obtains can also be comprised.Such as, in HBase database, by resolving Hlog, the data of respective table can be obtained (in HBase database, generally store data in the form of a table, by major key rowkey, be good for retrieve data also referred to as row), thus want migration data to write the timestamp (timestamp) of main cluster described in obtaining, the timestamp (timestamp) that can increase the main cluster of write before the former major key rowkey of data forms new major key rowkey, therefore, these newly-generated major keys and corresponding data can be write in the new table generated according to predetermined naming rule by writing module 330, namely in increment list, increment list can be used for inquiring about the former table of arbitrary period and be written with which data, i.e. incremental data.
According to the embodiment of the application, can also comprise: the step that mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.When mode of operation is transformed into fixed time segment data migration from described duration data migration by migration units, migration units suspends described lasting Data Migration pattern, and opens the Data Migration pattern of described fixed time section.
Described one or more migration units can also preserve the positional information of the part correspondence in described journal file be not processed in the journal file and described journal file be not processed when suspending described lasting Data Migration pattern, and the minimum time point in the record journal file that corresponding district management unit finally generates when suspending described lasting Data Migration pattern, that is, when described migration units by mode of operation from duration data migration model to the Data Migration patten transformation of fixed time section time, the journal file do not processed when preserving the Data Migration pattern suspended and continue, and the minimum time point recorded in the journal file that corresponding district management unit finally generates, when migration units by mode of operation from the Data Migration pattern recovery of fixed time section to the Data Migration pattern continued time, suspend the Data Migration pattern of described fixed time section, and according to the positional information of the part be not processed in the described journal file be not processed correspondence in described journal file and the work of the described lasting Data Migration pattern of described minimum time point recovery, namely, migration units continues the part be not processed in the journal file do not processed described in process, and the Data Migration restarting to continue according to described minimum time point (also has newly-generated with the described journal file wanting migration data corresponding from this minimum time point, Data Migration is proceeded) from this time point.
According to the embodiment of the application, can also comprise: described one or more migration units contrasts data in described main cluster and described standby cluster of specifying, to determine the conforming step of active and standby company-data.
The data of migration data and described standby cluster of specifying of wanting of resolving the acquisition of described journal file can be made comparisons, to determine the consistance of active and standby company-data by described migration units; Data in active and standby cluster can also be compared.Such as, in HBase cluster, (, only have and insert and delete two kinds of basic models in HBase when being in the operation of intercalation model), the data of resolving the acquisition of described journal file are made comparisons with the data that standby cluster is inquired about, due in HBase, most operation is all intercalation model, does not need like this to inquire about main cluster, alleviates the performance pressures of main cluster, reach again the object of reconciliation (that is, the contrast of the data consistency of active and standby cluster).When being in the operation of puncturing pattern, then the data of active and standby cluster are made comparisons simultaneously.Consistance comparing function can be dynamically open.
According to the embodiment of the application, can also comprise: described one or more migration units carries out migration table, increment list, migration units cluster switch, log path, many steps of carrying out dynamic-configuration for clustering functionality and consistance contrast switch of Data Migration for cluster to described appointment to described main cluster.Wherein, dynamic-configuration migration table can for recording the table wanting migration data, and increment list can be the table of recording increment data; Migration units cluster switch, according to the needs of actual conditions, can open and close migration units cluster; Log path (Hlog Path) is the store path of the journal file wanting migration data corresponding); Many standby clustering functionalitys, that is, increase one or more standby cluster; Consistance contrast switch, can for carrying out the switch of the opening and closing of consistance contrast.
According to the embodiment of the application, can also comprise: whether the state that in the described one or more migration units of main migration units monitoring in described one or more migration units, each migration units is run is normal.Mainly through collecting following index to monitor: the number of migration units, just in the number of the migration units of migration data, which district management unit is each migration units processing, the number of district management unit, also untreated Hlog sum, the number of each district management unit also untreated Hlog.
Due to the operation steps of the method for the application and the architectural feature of device substantially corresponding, can be cross-referenced, therefore not detailed part in the description of the present embodiment, see the related description in previous embodiment, can not repeat at this.
In one typically configuration, computing equipment comprises one or more processor (CPU), input/output interface, network interface and internal memory.
Internal memory may comprise the volatile memory in computer-readable medium, and the forms such as random access memory (RAM) and/or Nonvolatile memory, as ROM (read-only memory) (ROM) or flash memory (flashRAM).Internal memory is the example of computer-readable medium.
Computer-readable medium comprises permanent and impermanency, removable and non-removable media can be stored to realize information by any method or technology.Information can be computer-readable instruction, data structure, the module of program or other data.The example of the storage medium of computing machine comprises, but be not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic RAM (DRAM), the random access memory (RAM) of other types, ROM (read-only memory) (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc ROM (read-only memory) (CD-ROM), digital versatile disc (DVD) or other optical memory, magnetic magnetic tape cassette, tape magnetic rigid disk stores or other magnetic storage apparatus or any other non-transmitting medium, can be used for storing the information can accessed by computing equipment.According to defining herein, computer-readable medium does not comprise non-temporary computer readable media (transitory media), as data-signal and the carrier wave of modulation.
Also it should be noted that, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, commodity or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, commodity or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, commodity or the equipment comprising described key element and also there is other identical element.
Those skilled in the art should understand, the embodiment of the application can be provided as method, system or computer program.Therefore, the application can adopt the form of complete hardware embodiment, completely software implementation or the embodiment in conjunction with software and hardware aspect.And the application can adopt in one or more form wherein including the upper computer program implemented of computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) of computer usable program code.
The foregoing is only the embodiment of the application, be not limited to the application, for a person skilled in the art, the application can have various modifications and variations.Within all spirit in the application and principle, any amendment done, equivalent replacement, improvement etc., within the right that all should be included in the application.
Claims (24)
1. a system for Data Migration, for carrying out Data Migration between the main cluster and standby cluster of distributed data base, is characterized in that, comprise: main cluster and one or more standby cluster;
Wherein, described main cluster comprises one or more server, and each server has district management unit, and each district management unit generates at least one and the journal file wanting migration data relevant; Described journal file comprises: describedly want the content of migration data, described migration data of wanting writes the temporal information of described main cluster; And the one or more servers wherein, in described main cluster or standby cluster are configured with one or more migration units;
Described one or more migration units, for by resolving the journal file of wanting migration data relevant to described thus by the described standby cluster wanting migration data to write to specify.
2. system according to claim 1, is characterized in that, described one or more migration units and described one or more district management unit are configured on same station server correspondingly.
3. system according to claim 1, is characterized in that,
Described one or more migration units, wherein any one is main migration units, distributes to described one or more migration units process for what generated by described one or more district management unit to the described journal file wanting migration data relevant.
4. system according to claim 3, is characterized in that, each migration units in described one or more migration units comprises:
Whether monitoring module is normal for the running status monitoring each migration units in described one or more migration units when this migration units becomes main migration units.
5. system according to claim 1, is characterized in that, described migration units comprises:
Acquisition module is that generate to the described journal file wanting migration data relevant for obtaining corresponding district management unit under the Data Migration pattern continued;
Parsing module, for resolving described journal file, wants migration data described in obtaining; And
Writing module, for writing described migration data of wanting in the standby cluster of specifying.
6. system according to claim 5, is characterized in that,
Described acquisition module, it is that require to the described journal file wanting migration data relevant that the temporal information also generated for obtaining corresponding district management unit under the Data Migration pattern of at the appointed time section meets fixed time section;
Described parsing module, also meets for resolving described temporal information the journal file that described fixed time section requires, wants migration data with what obtain described fixed time section; And
Said write module, also for being write in the standby cluster of specifying by the migration data of wanting in described fixed time section.
7. the system according to claim 5 or 6, is characterized in that, said write module, also for resolving in the increment list of the incremental data write correspondence that described journal file obtains.
8. system according to claim 6, it is characterized in that, described migration units also comprises: mode conversion module, for the mode of operation of described migration units being changed between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.
9. system according to claim 8, is characterized in that, described mode conversion module comprises:
Transform subblock, for suspending described lasting Data Migration pattern, and opens the Data Migration pattern of described fixed time section.
10. system according to claim 9, is characterized in that, described mode conversion module also comprises:
Preserve submodule, for preserving the positional information of the part correspondence in described journal file be not processed in the journal file that has not been processed and described journal file when suspending described lasting Data Migration pattern; And
Record sub module, for recording the minimum time point when suspending described lasting Data Migration pattern in the journal file that finally generates of corresponding district management unit;
Described transform subblock, be further used for the Data Migration pattern suspending described fixed time section, and open described lasting Data Migration pattern according to the positional information corresponding in described journal file of the part be not processed in the described journal file be not processed and described minimum time point.
11. systems according to claim 1, is characterized in that, described migration units comprises: consistance contrast module, for contrasting data in described main cluster and described standby cluster of specifying, to determine the consistance of active and standby company-data.
12. systems according to claim 11, it is characterized in that, described migration units comprises: dynamic configuration module, carries out dynamic-configuration for carrying out the migration table of Data Migration, increment list, migration units cluster switch, log path, many standby clustering functionalitys and consistance contrast switch to described appointment for cluster to described main cluster.
The method of 13. 1 kinds of Data Migrations, for carrying out Data Migration between main cluster and standby cluster, is characterized in that, comprise:
One or more migration units is resolved and the journal file wanting migration data relevant;
Described migration data of wanting writes in the standby cluster of specifying by described one or more migration units;
Wherein, described main cluster comprises one or more server, and each server has district management unit, and each district management unit generates at least one and the described journal file wanting migration data relevant; Described journal file comprises: describedly want the content of migration data, described migration data of wanting writes the temporal information of described main cluster; Described one or more migration units is configured on the one or more servers in described main cluster or standby cluster.
14. methods according to claim 13, is characterized in that, one or more district management unit of described one or more migration units and described main cluster are configured on same station server correspondingly.
15. methods according to claim 13, is characterized in that, also comprise:
Main migration units in described one or more migration units, what generated by described one or more district management unit distributes to described one or more migration units to the described journal file wanting migration data relevant and processes, wherein, described main migration units is any one in described one or more migration units.
16. methods according to claim 15, is characterized in that, also comprise: whether the state that in the described one or more migration units of main migration units monitoring in described one or more migration units, each migration units is run is normal.
17. methods according to claim 13, is characterized in that,
Described one or more migration units is resolved and is comprised to the step of the described journal file wanting migration data relevant:
Under the Data Migration pattern continued, obtain corresponding district management unit that generate to the described journal file wanting migration data relevant;
Resolve described journal file, described in obtaining, want migration data;
The described step wanting migration data to write in the standby cluster of specifying comprises by described one or more migration units further:
Under the Data Migration pattern continued, described migration data of wanting is write in the standby cluster of specifying.
18. methods according to claim 17, is characterized in that,
Described one or more migration units is resolved and is comprised further to the step of the described journal file wanting migration data relevant:
Under the Data Migration pattern of at the appointed time section, obtaining temporal information that corresponding district management unit generates, to meet fixed time section that require to the described journal file wanting migration data relevant;
Resolve described temporal information and meet the journal file that described fixed time section requires, want migration data with what obtain described fixed time section; And
The described step wanting migration data to write in the standby cluster of specifying comprises by described one or more migration units further:
Under the Data Migration pattern of at the appointed time section, the migration data of wanting in described fixed time section is write in the standby cluster of specifying.
19. methods according to claim 17 or 18, is characterized in that, also comprise: described one or more migration units will be resolved in increment list corresponding to incremental data write that described journal file obtains.
20. methods according to claim 18, is characterized in that, also comprise: mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section.
21. methods according to claim 20, is characterized in that, mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section, comprising:
Suspend described lasting Data Migration pattern, and open the Data Migration pattern of described fixed time section.
22. methods according to claim 21, is characterized in that, mode of operation is changed by described one or more migration units between described lasting Data Migration pattern and the Data Migration pattern of described fixed time section, also comprises:
Preserve the positional information of the part correspondence in described journal file be not processed in the journal file and described journal file be not processed when suspending described lasting Data Migration pattern, and the minimum time point in the journal file of record last generation that corresponding district management unit generates when suspending described lasting Data Migration pattern;
Suspend the Data Migration pattern of described fixed time section, and open described lasting Data Migration pattern according to the positional information corresponding in described journal file of the part be not processed in the described journal file be not processed and described minimum time point.
23. methods according to claim 13, is characterized in that, also comprise: described one or more migration units contrasts data in described main cluster and described standby cluster of specifying, to determine the consistance of active and standby company-data.
24. methods according to claim 23, it is characterized in that, also comprise: described one or more migration units is carried out the migration table of Data Migration, increment list, migration units cluster switch, log path, many standby clustering functionality and consistance contrast switch to described appointment for cluster to described main cluster and carried out dynamic-configuration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310388988.9A CN104424283A (en) | 2013-08-30 | 2013-08-30 | Data migration system and data migration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310388988.9A CN104424283A (en) | 2013-08-30 | 2013-08-30 | Data migration system and data migration method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104424283A true CN104424283A (en) | 2015-03-18 |
Family
ID=52973263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310388988.9A Pending CN104424283A (en) | 2013-08-30 | 2013-08-30 | Data migration system and data migration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104424283A (en) |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105205154A (en) * | 2015-09-24 | 2015-12-30 | 浙江宇视科技有限公司 | Data migration method and device |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN105472045A (en) * | 2016-01-26 | 2016-04-06 | 北京百度网讯科技有限公司 | Database migration method and database migration device |
| CN105871987A (en) * | 2015-12-14 | 2016-08-17 | 乐视云计算有限公司 | High available system and method for data writing |
| CN105868343A (en) * | 2016-03-28 | 2016-08-17 | 上海携程商务有限公司 | Database migration method and system |
| CN106250432A (en) * | 2016-07-26 | 2016-12-21 | 浪潮软件股份有限公司 | A kind of hbase fault-tolerance approach based on persistence MQ |
| CN106294440A (en) * | 2015-05-27 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN106339279A (en) * | 2016-08-24 | 2017-01-18 | 杭州华三通信技术有限公司 | Service restoration method and device |
| CN106682021A (en) * | 2015-11-09 | 2017-05-17 | 五八同城信息技术有限公司 | Database migration method and device |
| CN106776876A (en) * | 2016-11-29 | 2017-05-31 | 用友网络科技股份有限公司 | Data migration method and data mover system |
| CN107122361A (en) * | 2016-02-24 | 2017-09-01 | 阿里巴巴集团控股有限公司 | Data mover system and method |
| CN107179879A (en) * | 2016-03-11 | 2017-09-19 | 伊姆西公司 | Method and apparatus for the Data Migration of storage device |
| WO2017167100A1 (en) * | 2016-03-31 | 2017-10-05 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN107247727A (en) * | 2017-04-28 | 2017-10-13 | 北京五八信息技术有限公司 | Data migration method and equipment |
| CN107391634A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107704550A (en) * | 2017-09-26 | 2018-02-16 | 深圳前海微众银行股份有限公司 | File migration method, apparatus and computer-readable recording medium |
| CN107888686A (en) * | 2017-11-14 | 2018-04-06 | 西南医科大学 | A kind of user data validation verification method positioned at HBase clients |
| WO2018072450A1 (en) * | 2016-10-17 | 2018-04-26 | Huawei Technologies Co., Ltd. | Method for elastic geographical database replication |
| CN108628874A (en) * | 2017-03-17 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data |
| CN109408280A (en) * | 2017-08-17 | 2019-03-01 | 北京金山云网络技术有限公司 | Data back up method, apparatus and system |
| CN109739690A (en) * | 2018-12-29 | 2019-05-10 | 平安科技（深圳）有限公司 | Backup method and Related product |
| WO2019095577A1 (en) * | 2017-11-14 | 2019-05-23 | 平安科技（深圳）有限公司 | Method and device for processing data request, computer apparatus, and storage medium |
| CN109947599A (en) * | 2019-03-25 | 2019-06-28 | 北京百度网讯科技有限公司 | Method and device is managed in more cluster management methods and device, cluster |
| CN110807013A (en) * | 2018-08-03 | 2020-02-18 | 阿里巴巴集团控股有限公司 | Data migration method and device for distributed data storage cluster |
| CN111400108A (en) * | 2020-06-09 | 2020-07-10 | 四川新网银行股份有限公司 | Method for building HBase disaster recovery cluster |
| CN112347189A (en) * | 2020-11-05 | 2021-02-09 | 江苏电力信息技术有限公司 | Cloud computing-based financial data consistency failure discovery and recovery method |
| CN109739690B (en) * | 2018-12-29 | 2024-05-28 | 平安科技（深圳）有限公司 | Backup method and related products |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102855314A (en) * | 2012-08-27 | 2013-01-02 | 济南大学 | Real-time database backup system and method based on FPGA (field programmable gate array) |
| CN103067433A (en) * | 2011-10-24 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method, device and system of data migration of distributed type storage system |
-
2013
- 2013-08-30 CN CN201310388988.9A patent/CN104424283A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103067433A (en) * | 2011-10-24 | 2013-04-24 | 阿里巴巴集团控股有限公司 | Method, device and system of data migration of distributed type storage system |
| CN102855314A (en) * | 2012-08-27 | 2013-01-02 | 济南大学 | Real-time database backup system and method based on FPGA (field programmable gate array) |
Non-Patent Citations (3)
| Title |
|---|
| FRANKLINSUN: "HBase技术简介", 《CSDN》 * |
| LARS GEORGE: "《HBase: The Definitive Guide》", 1 September 2011, O’REILLY MEDIA出版社 * |
| TERIY: "Hbase Replication介绍", 《CSDN》 * |
Cited By (40)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294440B (en) * | 2015-05-27 | 2019-06-07 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN106294440A (en) * | 2015-05-27 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN105205154A (en) * | 2015-09-24 | 2015-12-30 | 浙江宇视科技有限公司 | Data migration method and device |
| CN106682021A (en) * | 2015-11-09 | 2017-05-17 | 五八同城信息技术有限公司 | Database migration method and device |
| CN105404474A (en) * | 2015-12-07 | 2016-03-16 | 上海爱数信息技术股份有限公司 | Data migration method of heterogeneous distributed memory system |
| CN105871987A (en) * | 2015-12-14 | 2016-08-17 | 乐视云计算有限公司 | High available system and method for data writing |
| CN105472045A (en) * | 2016-01-26 | 2016-04-06 | 北京百度网讯科技有限公司 | Database migration method and database migration device |
| CN107122361A (en) * | 2016-02-24 | 2017-09-01 | 阿里巴巴集团控股有限公司 | Data mover system and method |
| CN107179879B (en) * | 2016-03-11 | 2020-04-03 | 伊姆西Ip控股有限责任公司 | Method and apparatus for data migration of storage device |
| CN107179879A (en) * | 2016-03-11 | 2017-09-19 | 伊姆西公司 | Method and apparatus for the Data Migration of storage device |
| CN105868343A (en) * | 2016-03-28 | 2016-08-17 | 上海携程商务有限公司 | Database migration method and system |
| CN105868343B (en) * | 2016-03-28 | 2019-05-21 | 上海携程商务有限公司 | Database migration method and system |
| WO2017167100A1 (en) * | 2016-03-31 | 2017-10-05 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| CN107291750A (en) * | 2016-03-31 | 2017-10-24 | 阿里巴巴集团控股有限公司 | A kind of data migration method and device |
| CN107291750B (en) * | 2016-03-31 | 2020-11-06 | 阿里巴巴集团控股有限公司 | Data migration method and device |
| US11940965B2 (en) | 2016-03-31 | 2024-03-26 | Alibaba Group Holding Limited | Data migration method and apparatus |
| US11354283B2 (en) | 2016-03-31 | 2022-06-07 | Alibaba Group Holding Limited | Data migration method and apparatus |
| CN106250432A (en) * | 2016-07-26 | 2016-12-21 | 浪潮软件股份有限公司 | A kind of hbase fault-tolerance approach based on persistence MQ |
| CN106339279A (en) * | 2016-08-24 | 2017-01-18 | 杭州华三通信技术有限公司 | Service restoration method and device |
| CN106339279B (en) * | 2016-08-24 | 2021-10-12 | 新华三技术有限公司 | Service recovery method and device |
| CN109804365A (en) * | 2016-10-17 | 2019-05-24 | 华为技术有限公司 | Elastic geography database copy method |
| WO2018072450A1 (en) * | 2016-10-17 | 2018-04-26 | Huawei Technologies Co., Ltd. | Method for elastic geographical database replication |
| CN106776876A (en) * | 2016-11-29 | 2017-05-31 | 用友网络科技股份有限公司 | Data migration method and data mover system |
| CN108628874A (en) * | 2017-03-17 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data |
| CN108628874B (en) * | 2017-03-17 | 2020-12-22 | 北京京东尚科信息技术有限公司 | Method and device for migrating data, electronic equipment and readable storage medium |
| CN107247727A (en) * | 2017-04-28 | 2017-10-13 | 北京五八信息技术有限公司 | Data migration method and equipment |
| CN107391634A (en) * | 2017-06-30 | 2017-11-24 | 北京奇虎科技有限公司 | Data migration method and device |
| CN109408280A (en) * | 2017-08-17 | 2019-03-01 | 北京金山云网络技术有限公司 | Data back up method, apparatus and system |
| CN107704550B (en) * | 2017-09-26 | 2021-04-30 | 深圳前海微众银行股份有限公司 | File migration method and device and computer readable storage medium |
| CN107704550A (en) * | 2017-09-26 | 2018-02-16 | 深圳前海微众银行股份有限公司 | File migration method, apparatus and computer-readable recording medium |
| CN107888686B (en) * | 2017-11-14 | 2020-10-02 | 西南医科大学 | User data validity verification method located at HBase client |
| WO2019095577A1 (en) * | 2017-11-14 | 2019-05-23 | 平安科技（深圳）有限公司 | Method and device for processing data request, computer apparatus, and storage medium |
| CN107888686A (en) * | 2017-11-14 | 2018-04-06 | 西南医科大学 | A kind of user data validation verification method positioned at HBase clients |
| CN110807013A (en) * | 2018-08-03 | 2020-02-18 | 阿里巴巴集团控股有限公司 | Data migration method and device for distributed data storage cluster |
| CN110807013B (en) * | 2018-08-03 | 2023-07-18 | 阿里巴巴集团控股有限公司 | Data migration method and device for distributed data storage cluster |
| CN109739690A (en) * | 2018-12-29 | 2019-05-10 | 平安科技（深圳）有限公司 | Backup method and Related product |
| CN109739690B (en) * | 2018-12-29 | 2024-05-28 | 平安科技（深圳）有限公司 | Backup method and related products |
| CN109947599A (en) * | 2019-03-25 | 2019-06-28 | 北京百度网讯科技有限公司 | Method and device is managed in more cluster management methods and device, cluster |
| CN111400108A (en) * | 2020-06-09 | 2020-07-10 | 四川新网银行股份有限公司 | Method for building HBase disaster recovery cluster |
| CN112347189A (en) * | 2020-11-05 | 2021-02-09 | 江苏电力信息技术有限公司 | Cloud computing-based financial data consistency failure discovery and recovery method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104424283A (en) | Data migration system and data migration method | |
| US11520670B2 (en) | Method and apparatus for restoring data from snapshots | |
| US10387673B2 (en) | Fully managed account level blob data encryption in a distributed storage environment | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| CN109407977B (en) | Big data distributed storage management method and system | |
| US8930364B1 (en) | Intelligent data integration | |
| US10628298B1 (en) | Resumable garbage collection | |
| US11397749B2 (en) | Asynchronous replication of in-scope table data | |
| CN103902410A (en) | Data backup acceleration method for cloud storage system | |
| CN104750755B (en) | A kind of data covering method and system after database active-standby switch | |
| US10694002B1 (en) | Data compression optimization based on client clusters | |
| CN107451172A (en) | Method of data synchronization and equipment for edition management system | |
| CN110022338A (en) | File reading, system, meta data server and user equipment | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| CN108319618A (en) | A kind of data distribution control method, system and the device of distributed memory system | |
| Pan et al. | Navigating the landscape of distributed file systems: Architectures, implementations, and considerations | |
| CN113254460B (en) | Data processing method, system, electronic device and storage medium | |
| CN108234566B (en) | Cluster data processing method and device | |
| US11436089B2 (en) | Identifying database backup copy chaining | |
| US9037762B2 (en) | Balancing data distribution in a fault-tolerant storage system based on the movements of the replicated copies of data | |
| US11334456B1 (en) | Space efficient data protection | |
| CN115955488A (en) | Distributed storage copy cross-computer room placement method and device based on copy redundancy | |
| KR102089710B1 (en) | Continous data mangement system and method | |
| CN114089923A (en) | Double-live storage system and data processing method thereof | |
| US9436407B1 (en) | Cursor remirroring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150318 |
|
| RJ01 | Rejection of invention patent application after publication |