CN108628874A - Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data - Google Patents
Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data Download PDFInfo
- Publication number
- CN108628874A CN108628874A CN201710159838.9A CN201710159838A CN108628874A CN 108628874 A CN108628874 A CN 108628874A CN 201710159838 A CN201710159838 A CN 201710159838A CN 108628874 A CN108628874 A CN 108628874A
- Authority
- CN
- China
- Prior art keywords
- cluster
- snapshot
- data
- source
- copy services
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of method, apparatus of migrating data, electronic equipment and readable storage medium storing program for executing, can solve in data migration process, and service application stops writing problem for a long time, to realize seamless migration data that service application does not stop to write.This method includes：Copy services between configuration cluster, and suspend copy services；The snapshot of the data at the source cluster current time is created, and the snapshot is exported into target cluster；Restart copy services after update is completed using the data of target cluster described in the snapshot update；Operation in the target cluster during playback pause copy services about service application to source cluster, and target cluster is written into the operation.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of method, apparatus of migrating data, electronic equipments and can
Read storage medium.
Background technology
Since data volume increases, big data is universal, more and more operation systems are selected using data-base cluster as depositing
Storage.With the expansion of the quantity and scale of cluster, Data Migration is on the increase between the cluster being related to.
Data Migration is typically that (Distcp, i.e. distributed copy are for big using Distcp between the cluster of the prior art
The tool copied between scale cluster internal and cluster) modes of tool copied files carries out, and executes data in target cluster
Migration is realized in load.By taking a kind of HBase (distributed, towards row PostgreSQL database) cluster as an example, the migration of the prior art
The process of data is substantially as shown in Figure 1.
In realizing process of the present invention, inventor has found that at least there are the following problems in the prior art：Distcp tools exist
Forbidden list is needed before operation, it is ensured that be written without data.Therefore, service application needs to stop for a long time in data migration process
It writes.This will have an impact the use of business on line, form bad user experience, while also cannot be satisfied service application pair
The requirement of data-base cluster High Availabitity.
Invention content
In view of this, the embodiment of the present invention provides a kind of method, apparatus of migrating data, electronic equipment and readable storage medium
Matter can solve in data migration process, and service application stops writing problem for a long time, to realize that it is seamless that service application does not stop to write
Migrating data.
To achieve the above object, one side according to the ... of the embodiment of the present invention provides a kind of method of migrating data.
A kind of method of migrating data of the embodiment of the present invention includes：Copy services between configuration cluster, and suspend duplication clothes
Business, wherein the copy services are will to be sent to target cluster, the playback of target cluster to the operation of source cluster about service application
Simultaneously target cluster is written in the operation by the operation, and the pause copy services refer to retained business using the behaviour to source cluster
Work wouldn't be sent to target cluster；The snapshot of the data at the source cluster current time is created, and the snapshot is exported into mesh
Mark cluster；Restart copy services after update is completed using the data of target cluster described in the snapshot update；In the mesh
The operation during playback pause copy services about service application to source cluster in cluster is marked, and object set is written into the operation
Group.
Optionally, the cluster is HBase clusters.
Optionally, copy services include between configuring cluster：Replication queues are configured, by the WAL days of the source cluster
Will is sent to the target cluster by Replication queues, wherein the WAL daily records are for preserving service application to source
The operation of cluster；And the WAL daily records are played back in the target cluster, more to the operation of the source cluster by service application
Newly to target cluster.
Optionally, the method further includes：During suspending copy services, the source cluster retains the WAL daily records.
Optionally, include using the data of target cluster described in the snapshot update：Utilize mesh described in the snapshot update
Mark the definition of the table of cluster；Restore the Region information of table；And the Region of offline variation, update the information of meta tables.
To achieve the above object, other side according to the ... of the embodiment of the present invention provides a kind of device of migrating data.
A kind of device of migrating data of the embodiment of the present invention includes：Configuration module, for configuring copy services between cluster,
And suspend copy services, wherein the copy services are will to be sent to target cluster to the operation of source cluster about service application,
Simultaneously target cluster is written in the operation by the target cluster playback operation, and the pause copy services refer to retained business application
Target cluster wouldn't be sent to the operation of source cluster；Snapshot module, the data for creating source cluster current time
Snapshot, and the snapshot is exported into target cluster；Update module, for the number using target cluster described in the snapshot update
According to, update complete after, restart copy services；Replication module, for the playback pause copy services phase in the target cluster
Between operation about service application to source cluster, and target cluster is written into the operation.
Optionally, the cluster is HBase clusters.
Optionally, the configuration module is additionally operable to：Replication queues are configured, the WAL daily records of the source cluster are led to
It crosses Replication queues and is sent to the target cluster, wherein the WAL daily records are for preserving service application to source cluster
Operation；And the WAL daily records are played back in the target cluster, service application is updated to the operation of the source cluster
Target cluster.
Optionally, the configuration module is additionally operable to：During suspending copy services, the source cluster retains described WAL days
Will.
Optionally, the update module is additionally operable to：Utilize the definition of the table of target cluster described in the snapshot update；Restore
The Region information of table；And the Region of offline variation, update the information of meta tables.
To achieve the above object, according to the ... of the embodiment of the present invention in another aspect, providing a kind of electronic equipment.
The a kind of electronic equipment of the embodiment of the present invention includes：At least one processor；And at least one processing
The memory of device communication connection；Wherein,
The memory is stored with the instruction that can be executed by one processor, and described instruction is by least one place
It manages device to execute, so that the method that at least one processor is able to carry out the migrating data of the embodiment of the present invention.
To achieve the above object, another aspect according to the ... of the embodiment of the present invention, it is readable to provide a kind of non-transient computer
Storage medium.
A kind of non-transient computer readable storage medium of the embodiment of the present invention stores computer instruction, and the computer refers to
The method for enabling the migrating data for making the computer execute the embodiment of the present invention.
One embodiment in foregoing invention has the following advantages that or advantageous effect：It is combined by using snapshot and duplication
Mode so that data migration process service application need not stop to write, and then is realized and is not generating shadow to service application
The seamless migration of finishing service application data in the case of sound；By the way that during suspending copy services, WAL daily records are temporarily overstock
In Replication queues, so that the WAL daily records not replicated will not delete, and after restarting copy services, it can
Overstocked data manipulation is set to continue to be consumed and complete to replicate；By creating the online snapshot of table to be migrated on the cluster of source, reflect
The read-write of source cluster table is not interfered in creating snapshot, and it is HDFS levels to export snapshot, so as to ensure using soon
During according to migrating data, the influence of clustering performance will be minimized；By after the completion of snapshot data migrates, opening again
The copy services of source cluster are opened, cluster Replication queues overstocked WAL daily records in source during consumption migrates snapshot are completed to increase
The migration of data is measured, so that source cluster reaches final consistent with the data of target cluster.
Further effect possessed by above-mentioned non-usual optional mode adds hereinafter in conjunction with specific implementation mode
With explanation.
Description of the drawings
Attached drawing does not constitute inappropriate limitation of the present invention for more fully understanding the present invention.Wherein：
Fig. 1 is the flow diagram of the method for the Data Migration of the prior art；
Fig. 2 is a kind of schematic diagram of the key step of the method for migrating data according to the ... of the embodiment of the present invention；
Fig. 3 is a kind of main flow schematic diagram of the method for migrating data according to the ... of the embodiment of the present invention；
Fig. 4 is a kind of schematic diagram of the main modular of the device of migrating data according to the ... of the embodiment of the present invention；
Fig. 5 is a kind of hardware architecture diagram of the electronic equipment of migrating data according to the ... of the embodiment of the present invention；
Fig. 6 diagrammatically illustrates the computer that terminal device or server according to the embodiment of the present invention may be implemented
The structural schematic diagram of system.
Specific implementation mode
It explains to the exemplary embodiment of the present invention below in conjunction with attached drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
The description to known function and structure is omitted for clarity and conciseness in sample in following description.
In order to solve in prior art data migration process, service application needs to stop to write problem for a long time, and the present invention is implemented
Example provides a kind of technical solution of migrating data, realizes that service application does not stop the number write snapshot and by way of replicating combination
According to seamless migration.
In the embodiment of the present invention, service application may be mounted on terminal device, by terminal device and source database collection
Group is communicated, and when communication, used network may include various connection types, such as wired, wireless communication link or light
Fiber-optic cable etc..
Wherein, terminal device can be the various electronic equipments with display screen and supported web page browsing, including but not
It is limited to smart mobile phone, tablet computer, E-book reader, MP3 player (Moving Picture Experts Group
Audio Layer III, dynamic image expert's compression standard audio level 3), MP4 (Moving Picture Experts
Group Audio Layer IV, dynamic image expert's compression standard audio level 4) player, pocket computer on knee and
Desktop computer etc..Various telecommunication customer end applications, such as web browser applications can be installed on terminal device.
Fig. 2 is a kind of schematic diagram of the key step of the method for migrating data according to the ... of the embodiment of the present invention.
As shown in Fig. 2, a kind of method of migrating data of the embodiment of the present invention mainly includes the following steps：
Step S21：Copy services between configuration cluster, and suspend copy services, wherein the copy services are will be about industry
Business application is sent to target cluster to the operation of source cluster, and simultaneously object set is written in the operation by the target cluster playback operation
Group, the pause copy services refer to retained business using that wouldn't be sent to target cluster to the operation of source cluster, wait restarting and answer
Uniform is engaged in and then the operation is sent to target cluster, to complete to replicate.
The purpose of this step is the duplication of data between realization cluster.There are addition, modification, deletion in source database cluster
When Deng operating, which can be recorded, such as in HBase clusters, and WAL daily records can be written (i.e. by RegionServer in operation
Write-Ahead-Log, and writes log system, is a kind of efficient daily record algorithm in database.Such as in HBase clusters,
It is a kind of daily record that RegionServer is used for recording operation content during handling data and being inserted into and delete).Pass through
Toward object set pocket transmission WAL daily records, and after target cluster receives, WAL daily records write-in object set is played back in the form of client
Group, to complete to operate the duplication of targeted data.
After step S21 sets up copy services between cluster and suspends, the migration of data is proceeded by from step S22.
Step S22：The snapshot of the data at the source cluster current time is created, and the snapshot is exported into object set
Group.By exporting the online snapshot of source company-data, the total data of the source cluster at a certain moment is migrated.
Step S23：Restart copy services after update is completed using the data of target cluster described in the snapshot update.
After step S22 is created and exports snapshot, by target cluster recovery snapshot, realizing and utilizing snapshot update target cluster
The purpose of data.
Step S24：It is played back in the target cluster during suspending copy services about service application to the behaviour of source cluster
Make, and target cluster is written into the operation, to complete the seamless migration of data.The purpose of this step is in target cluster
The operation about service application to source cluster overstock during consumption migration snapshot, is completed source cluster increment during snapshot migration
The migration of data.
Wherein, the cluster in the embodiment of the present invention can be, but not limited to as HBase clusters, in addition it is also possible to be other classes
Type data-base cluster.In addition, Replication queues can be utilized between aforementioned arrangements cluster the step of copy services
(Replication is a kind of mechanism of company-data real-time synchronization) is realized.In HBase clusters, source cluster and target are established
Replication relationships between cluster utilize the WAL daily records of the operation of source cluster about service application by that will record
Replication queues are sent to target cluster, and carry out the playback of WAL daily records in target cluster, so as to realize source collection
Group's purpose synchronous with the data of target cluster.
In the embodiment of the present invention, after configuration copy services, it is also necessary to suspend copy services, first carry out source cluster full dose number
According to snapshot migration, by by the total data of current time source cluster in the way of snapshot derived from target cluster.
It (creates source cluster snapshot to start to stop when updating completion data to target cluster) during migrating due to snapshot, still may be used
Service application can be had and the operations such as increased, deleted, changed to source cluster, therefore, copy services will be suspended during the migration of this snapshot, and
The WAL daily records generated during copy services will be suspended to overstock in advance, be not sent to target cluster.Wait for that snapshot migration is completed, i.e., it is sharp
The data of target cluster are completed with snapshot update and then restart copy services, will be migrated the WAL daily records overstock during snapshot and are sent out
It send to target cluster, completes the migration of source cluster incremental data, to realize the seamless migration of data between data-base cluster.
In addition, in the embodiment of the present invention, mainly comprised the following processes using the data of snapshot update target cluster：Using institute
State the definition of the table of target cluster described in snapshot update；(Region is the storage of HBase data and management to the Region of recovery table
Base unit.Can include one or more Region in one table) information；And the Region of offline variation, update meta
The information of table.
Fig. 3 is a kind of main flow schematic diagram of the method for migrating data according to the ... of the embodiment of the present invention.Below according to Fig. 3
Flow describe in detail to the method for the migrating data of the embodiment of the present invention.
It is evidenced from the above discussion that the embodiment of the present invention realizes that data seamless migrates snapshot and by way of replicating combination.
Basic principle is the online snapshot by exporting source company-data, migrates the total data of a certain time data library table, in conjunction with
The mechanism of duplication completes the migration of incremental data, reach source cluster with target company-data final consistent, entire transition process
Stop writing without application.
It is specific as follows：
One, data replication service between configuration cluster, and first suspend copy services
Configuration copy services are simultaneously first suspended, the WAL daily records that the WAL daily records of source cluster can temporarily overstock, and not replicate at this time
It will not delete.Copy services are reopened after the completion of export snapshot, overstocked data will continue to consume and complete to replicate.This makes collection
Data can reach final consistent between group.
Concrete operations：1. establishing the copy services relationship between source cluster and target cluster, wherein target cluster is source collection
The slave cluster of group configuration copy services；2. executing pause copy services order.
The principle of data replication service：By taking HBase clusters as an example, when service application is inserted into HBase or deletes data,
Whether insert or delete operation can be written WAL daily records in the form of it can play back and (wherein, write WAL daily records and match by RegionServer
Replication is set not to be associated with.Whether writing WAL daily records, to be parameter can control, and acquiescence is opened), after opening copy services,
The WAL daily records of record operation can be put into Replication queues by source cluster, and by WAL daily records asynchronous transmission to object set
Group.WAL daily records can be retained in the cluster of source until copy services are completed.
Two, it creates online snapshot and exports snapshot, migrating data
The online snapshot (snapshot) of data to be migrated is created on the cluster of source, is created snapshot and is not interfered with source cluster table
Read-write, snapshot is actually a series of metadata information set, at this time without copy data.Export is executed again shines mesh soon
Mark cluster completes the migration of data in target cluster recovery snapshot after the completion of export.It is specific as follows：
1. cluster creates snapshot in source.Creating snapshot includes：Execute online snapshot, the description information of backup table, by dividing
Cloth affairs, the Region information of backup table create the reference document etc. of HFile；
2. snap copy is exported snapshot to target cluster.All HBase table snapshot numbers that snapshot can be related at this time
According to copying target cluster to.Wherein, export snapshot is that the ExportSnapshot (export snapshot) carried by HBase is realized,
Essence is to carry out the copy of table snapshot data by executing MapReduce modes.In addition, since copy data are HDFS
(Hadoop Distributed File System, i.e. Hadoop distributed file systems, it provides high-throughput and is answered to access
With the data of program, those is suitble to have the application program of super large data set) level, so influence to clustering performance compared with
It is small；
3. being executed in target cluster and restoring snapshot restore_snapshot orders, the update of target company-data is completed,
Realize snapshot data migration.It is described for restoring snapshot in HBase clusters, mainly divides the following steps：
(1) definition of table is updated：First determine whether that table whether there is, there is no then create table.Then judge the definition letter of table
Whether breath has update, has update then to be defined with the table of snapshot and covers current table definition.
(2) restore Region：Region is compared, is restored one by one with the Region information in snapshot.To not have in snapshot
Have, existing Region is deleted now；By what snapshot and present Region comparisons changed restore；To have in snapshot, it is existing
Without Region created.
(3) Region of offline variation updates the information of meta tables.
Three, restore copy services
After the completion of snapshot data migration, so that it may to reopen the copy services of source cluster, restart copy services opisthogenesis
Cluster continues toward object set pocket transmission WAL daily records, and target cluster can consume source cluster during (i.e. aforementioned " playback ") migration snapshot
Overstocked WAL daily records, copy data to target cluster.This operation can execute always, sent until WAL daily records are whole,
Replication queues are not overstock.So far, source cluster just can reach final consistent with the table data of target cluster, entirely move
It moves past journey and does not need business side using stopping writing, realize the seamless migration of data.
It is noted that in the embodiment of the present invention, aims within WAL days not being sent to before target cluster, remain in source
In cluster.If being successfully sent to target cluster, main cluster WAL daily records no longer retain.
The method of migrating data according to the ... of the embodiment of the present invention can be seen that by using snapshot and the duplication side of being combined
Formula is realized so that data migration process service application need not stop to write in the feelings not had an impact to service application
The seamless migration of finishing service application data under condition；By the way that during suspending copy services, WAL daily records are temporarily overstock
In Replication queues, so that the WAL daily records not replicated will not delete, and after restarting copy services, it can make
Overstocked data manipulation continuation is consumed and completes to replicate；By creating the online snapshot of table to be migrated on the cluster of source, in view of
The read-write that snapshot does not interfere with source cluster table is created, and it is HDFS levels to export snapshot, and snapshot is being utilized so as to ensure
During migrating data, the influence of clustering performance will be minimized；By after the completion of snapshot data migrates, reopening
The copy services of source cluster, the overstocked WAL daily records of source cluster Replication queues, complete increment during consumption migrates snapshot
The migration of data, so that source cluster reaches final consistent with the data of target cluster.
Fig. 4 is a kind of schematic diagram of the main modular of the device of migrating data according to the ... of the embodiment of the present invention.
As shown in figure 4, a kind of device 40 of migrating data of the embodiment of the present invention includes mainly following module：Configuration module
401, snapshot module 402, update module 403, replication module 404, wherein：
Configuration module 401 for configuring copy services between cluster, and suspends copy services, wherein the copy services are
Target cluster will be sent to the operation of source cluster about service application, the target cluster playback operation simultaneously writes the operation
Entering target cluster, the pause copy services refer to retained business using wouldn't be sent to target cluster to the operation of source cluster,
Copy services to be restarted and then by it is described operation be sent to target cluster, to complete to replicate；Snapshot module 402, for creating
The snapshot of the data at the source cluster current time, and the snapshot is exported into target cluster；Update module 403, for profit
The data of target cluster described in the snapshot update restart copy services after update is completed；Replication module 404 is used for
Operation in the target cluster during playback pause copy services about service application to source cluster, and the operation is written
Target cluster, to complete the seamless migration of data.
Wherein, the cluster in the embodiment of the present invention can be, but not limited to as HBase clusters.
In addition, configuration module 401 can be additionally used in：Replication queues are configured, the WAL daily records of the source cluster are led to
It crosses Replication queues and is sent to the target cluster, wherein the WAL daily records are for preserving service application to source cluster
Operation；And the WAL daily records are played back in the target cluster, service application is updated to the operation of the source cluster
Target cluster.
In addition, configuration module 401 can be additionally used in：During suspending copy services, the source cluster retains described WAL days
Will.
In the embodiment of the present invention, update module 403 can be additionally used in：Utilize the table of target cluster described in the snapshot update
Definition；Restore the Region information of table；And the Region of offline variation, update the information of meta tables.
The device of migrating data according to the ... of the embodiment of the present invention can be seen that through the cooperation between each module, using fast
It is combined mode according to duplication, so that data migration process service application need not stop to write, realization is answered to business
With the seamless migration of finishing service application data in the case of not having an impact；By during suspending copy services, by WAL days
Will is temporarily overstock in Replication queues, so that the WAL daily records not replicated will not delete, and replicates clothes restarting
After business, overstocked data manipulation can be made to continue to be consumed and complete to replicate；By creating table to be migrated on the cluster of source
Online snapshot does not interfere with the read-write of source cluster table in view of snapshot is created, and it is HDFS levels to export snapshot, so as to protect
Barrier will minimize the influence of clustering performance during using snapshot migrating data；By having been migrated in snapshot data
Cheng Hou reopens the copy services of source cluster, cluster Replication queues overstocked WAL in source during consumption migrates snapshot
The migration of incremental data is completed in daily record, so that source cluster reaches final consistent with the data of target cluster.
According to an embodiment of the invention, the present invention also provides a kind of electronic equipment and a kind of readable storage medium storing program for executing.
The electronic equipment of the embodiment of the present invention includes：At least one processor；And it is logical at least one processor
Believe the memory of connection；Wherein, the memory is stored with the instruction that can be executed by one processor, and described instruction is by institute
It states at least one processor to execute, so that the method that at least one processor executes migrating data provided by the present invention.
The non-transient computer readable storage medium of the embodiment of the present invention stores computer instruction, and the computer instruction is used
In the method for making the computer execute migrating data provided by the present invention.
The hardware architecture diagram of the electronic equipment of the method for Fig. 5 migrating datas according to the ... of the embodiment of the present invention.Such as Fig. 5 institutes
Show, which includes：One or more processors 51 and memory 52, in Fig. 5 by taking a processor 51 as an example.Its
In, memory 52 is non-transient computer readable storage medium provided by the present invention.
The electronic equipment of the method for migrating data can also include：Input unit 53 and output device 54.
Processor 51, memory 52, input unit 53 can be connected with output device 54 by bus or other modes,
In Fig. 5 for being connected by bus.
Memory 52 is used as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, non-
Transient computer executable program and module, as the corresponding program instruction of the method for the migrating data in the embodiment of the present invention/
Module (for example, attached configuration module shown in Fig. 4 401, snapshot module 402, update module 403, replication module 404).Processor
51 are stored in non-transient software program, instruction and module in memory 52 by operation, so that execute server is various
Application of function and data processing, that is, the method for realizing the migrating data in above method embodiment.
Memory 52 may include storing program area and storage data field, wherein storing program area can storage program area,
At least one required application program of function；Storage data field can be stored to be created according to using for the device of migrating data
Data etc..In addition, memory 52 may include high-speed random access memory, can also include non-transient memory, such as extremely
A few disk memory, flush memory device or other non-transient solid-state memories.In some embodiments, memory 52
Optional includes the memory remotely located relative to processor 51, these remote memories can pass through network connection to transport number
According to device.The example of above-mentioned network includes but not limited to internet, intranet, LAN, mobile radio communication and its group
It closes.
Input unit 53 can receive the number or character information of input, and generates and set with the user of the device of migrating data
It sets and the related key signals of function control inputs.Output device 54 may include that display screen etc. shows equipment.
One or more of modules are stored in the memory 52, when by one or more of processors 51
When execution, the method that executes the migrating data in above-mentioned any means embodiment.
The said goods can perform the method that the embodiment of the present invention is provided, and has the corresponding function module of execution method and has
Beneficial effect.The not technical detail of detailed description in the present embodiment, reference can be made to the method that the embodiment of the present invention is provided.
Below with reference to Fig. 6, it illustrates the calculating suitable for terminal device or server for realizing the embodiment of the present invention
The structural schematic diagram of machine system 600.
As shown in fig. 6, computer system 600 includes central processing module (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into the program in random access storage device (RAM) 603 from storage section 608 and
Execute various actions appropriate and processing.In RAM603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM602 and RAM603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to bus
604。
It is connected to I/O interfaces 605 with lower component：Importation 606 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 608 including hard disk etc.；
And the communications portion 609 of the network interface card including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net executes communication process.Driver 610 is also according to needing to be connected to I/O interfaces 605.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 610, as needed in order to be read from thereon
Computer program be mounted into storage section 608 as needed.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be tangibly embodied in machine readable
Computer program on medium, the computer program include the program code for method shown in execution flow chart.At this
In the embodiment of sample, which can be downloaded and installed by communications portion 609 from network, and/or from removable
Medium 611 is unloaded to be mounted.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for a part for one module, program segment, or code of table, the module, program segment, or code includes one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it wants
It is noted that the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart, Ke Yiyong
The dedicated hardware based system of defined functions or operations is executed to realize, or can be referred to specialized hardware and computer
The combination of order is realized.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be arranged in the processor, for example, can be described as：A kind of processor packet
Include configuration module, snapshot module, update module, replication module.Wherein, the title of these modules is not constituted under certain conditions
Restriction to the module itself, for example, configuration module is also described as " configuring copy services between cluster, and suspending duplication
The module of service ".
From the above, it can be seen that mode is combined by using snapshot and duplication, so that Data Migration
Process service application need not stop to write, and realize the nothing of the finishing service application data in the case where not had an impact to service application
Seam migration；By during suspending duplication, WAL daily records temporarily being overstock in Replication queues, so that not replicating
WAL daily records will not delete, and after restarting copy services, can make overstocked data manipulation continue to be consumed and completed it is multiple
System；By creating the online snapshot of table to be migrated on the cluster of source, the read-write of source cluster table is not interfered in view of establishment snapshot, and
It is HDFS levels to export snapshot, will be to the shadow of clustering performance so as to ensure during using snapshot migrating data
Sound minimizes；By after the completion of snapshot data migrates, reopening the copy services of source cluster, during consumption migrates snapshot
The overstocked WAL daily records of source cluster Replication queues, complete the migration of incremental data, so that source cluster and mesh
The data of mark cluster reach final consistent.
Above-mentioned specific implementation mode, does not constitute limiting the scope of the invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made by within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (12)
1. a kind of method of migrating data, which is characterized in that including：
Configure copy services between cluster, and suspend copy services, wherein the copy services be by about service application to source collection
The operation of group is sent to target cluster, and simultaneously target cluster, the pause is written in the operation by the target cluster playback operation
Copy services refer to retained business using wouldn't be sent to target cluster to the operation of source cluster；
The snapshot of the data at the source cluster current time is created, and the snapshot is exported into target cluster；
Restart copy services after update is completed using the data of target cluster described in the snapshot update；
The operation about service application to source cluster during playback pause copy services in the target cluster, and by the behaviour
Make write-in target cluster.
2. according to the method described in claim 1, it is characterized in that, the cluster is HBase clusters.
3. according to the method described in claim 1, it is characterized in that, copy services include between configuration cluster：
Replication queues are configured, the WAL daily records of the source cluster are sent to the mesh by Replication queues
Mark cluster, wherein the WAL daily records are for preserving operation of the service application to source cluster；And
The WAL daily records are played back in the target cluster, service application is updated to object set to the operation of the source cluster
Group.
4. according to the method described in claim 3, it is characterized in that, the method further includes：During suspending copy services, institute
It states source cluster and retains the WAL daily records.
5. according to the method described in claim 1, it is characterized in that, using target cluster described in the snapshot update data packet
It includes：
Utilize the definition of the table of target cluster described in the snapshot update；
Restore the Region information of table；And
The Region of offline variation updates the information of meta tables.
6. a kind of device of migrating data, which is characterized in that including：
Configuration module for configuring copy services between cluster, and suspends copy services, wherein the copy services be by about
Service application is sent to target cluster to the operation of source cluster, and simultaneously target is written in the operation by the target cluster playback operation
Cluster, the pause copy services refer to retained business using wouldn't be sent to target cluster to the operation of source cluster；
Snapshot module, the snapshot of the data for creating the source cluster current time, and the snapshot is exported into object set
Group；
Update module after update is completed, restarts duplication clothes for the data using target cluster described in the snapshot update
Business；
Replication module, for being played back in the target cluster during suspending copy services about service application to the behaviour of source cluster
Make, and target cluster is written into the operation.
7. device according to claim 6, which is characterized in that the cluster is HBase clusters.
8. device according to claim 6, which is characterized in that the configuration module is additionally operable to：
Replication queues are configured, the WAL daily records of the source cluster are sent to the mesh by Replication queues
Mark cluster, wherein the WAL daily records are for preserving operation of the service application to source cluster；And
The WAL daily records are played back in the target cluster, service application is updated to object set to the operation of the source cluster
Group.
9. device according to claim 8, which is characterized in that the configuration module is additionally operable to：In the pause copy services phase
Between, the source cluster retains the WAL daily records.
10. device according to claim 6, which is characterized in that the update module is additionally operable to：
Utilize the definition of the table of target cluster described in the snapshot update；
Restore the Region information of table；And
The Region of offline variation updates the information of meta tables.
11. a kind of electronic equipment, which is characterized in that including：
At least one processor；And
The memory being connect at least one processor communication；Wherein,
The memory is stored with the instruction that can be executed by one processor, and described instruction is by least one processor
It executes, so that at least one processor is able to carry out the method described in any one of claim 1-5.
12. a kind of non-transient computer readable storage medium, which is characterized in that the non-transient computer readable storage medium is deposited
Store up computer instruction, the method that the computer instruction is used to that the computer perform claim to be made to require described in any one of 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710159838.9A CN108628874B (en) | 2017-03-17 | 2017-03-17 | Method and device for migrating data, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710159838.9A CN108628874B (en) | 2017-03-17 | 2017-03-17 | Method and device for migrating data, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108628874A true CN108628874A (en) | 2018-10-09 |
| CN108628874B CN108628874B (en) | 2020-12-22 |
Family
ID=63687684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710159838.9A Active CN108628874B (en) | 2017-03-17 | 2017-03-17 | Method and device for migrating data, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108628874B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209653A (en) * | 2019-06-04 | 2019-09-06 | 中国农业银行股份有限公司 | HBase data migration method and moving apparatus |
| CN111241060A (en) * | 2020-01-08 | 2020-06-05 | 苏州科达科技股份有限公司 | Data migration method, system, device and storage medium |
| CN111459913A (en) * | 2020-03-31 | 2020-07-28 | 北京金山云网络技术有限公司 | Capacity expansion method and device of distributed database and electronic equipment |
| CN111538719A (en) * | 2020-04-30 | 2020-08-14 | 深圳前海微众银行股份有限公司 | Data migration method, device, equipment and computer storage medium |
| CN112069152A (en) * | 2020-09-08 | 2020-12-11 | 北京达佳互联信息技术有限公司 | Database cluster upgrading method, device, equipment and storage medium |
| CN112463762A (en) * | 2020-11-06 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for cross-cluster real-time data migration and synchronization |
| CN112527767A (en) * | 2020-12-03 | 2021-03-19 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restart of distributed database |
| CN112631994A (en) * | 2020-12-29 | 2021-04-09 | 深圳市商汤科技有限公司 | Data migration method and system |
| CN113032704A (en) * | 2021-02-24 | 2021-06-25 | 广州虎牙科技有限公司 | Data processing method, device, electronic equipment and medium |
| CN113377763A (en) * | 2020-03-10 | 2021-09-10 | 阿里巴巴集团控股有限公司 | Database table switching method and device, electronic equipment and computer storage medium |
| CN113438275A (en) * | 2021-05-27 | 2021-09-24 | 众安在线财产保险股份有限公司 | Data migration method and device, storage medium and data migration equipment |
| CN113742422A (en) * | 2021-08-20 | 2021-12-03 | 广州市易工品科技有限公司 | Data synchronization accuracy verification method and device |
| CN113946293A (en) * | 2021-10-27 | 2022-01-18 | 北京达佳互联信息技术有限公司 | Cluster data migration method and device, electronic equipment and storage medium |
| CN114546989A (en) * | 2022-02-22 | 2022-05-27 | 重庆长安汽车股份有限公司 | Hbase incremental data migration system, method and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876883A (en) * | 2009-11-30 | 2010-11-03 | 英业达股份有限公司 | Method for keeping remote operation of virtual machine uninterrupted |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN102917072A (en) * | 2012-10-31 | 2013-02-06 | 北京奇虎科技有限公司 | Device, system and method for carrying out data migration between data server clusters |
| CN104424283A (en) * | 2013-08-30 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Data migration system and data migration method |
| CN105607954A (en) * | 2015-12-21 | 2016-05-25 | 华南师范大学 | Stateful container online migration method and apparatus |
| CN105718570A (en) * | 2016-01-20 | 2016-06-29 | 北京京东尚科信息技术有限公司 | Data migration method and device used for database |
| US20160196324A1 (en) * | 2015-01-05 | 2016-07-07 | Iguazio Systems Ltd. | Service oriented data management and architecture |
-
2017
- 2017-03-17 CN CN201710159838.9A patent/CN108628874B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101876883A (en) * | 2009-11-30 | 2010-11-03 | 英业达股份有限公司 | Method for keeping remote operation of virtual machine uninterrupted |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN102917072A (en) * | 2012-10-31 | 2013-02-06 | 北京奇虎科技有限公司 | Device, system and method for carrying out data migration between data server clusters |
| CN104424283A (en) * | 2013-08-30 | 2015-03-18 | 阿里巴巴集团控股有限公司 | Data migration system and data migration method |
| US20160196324A1 (en) * | 2015-01-05 | 2016-07-07 | Iguazio Systems Ltd. | Service oriented data management and architecture |
| CN105607954A (en) * | 2015-12-21 | 2016-05-25 | 华南师范大学 | Stateful container online migration method and apparatus |
| CN105718570A (en) * | 2016-01-20 | 2016-06-29 | 北京京东尚科信息技术有限公司 | Data migration method and device used for database |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110209653B (en) * | 2019-06-04 | 2021-11-23 | 中国农业银行股份有限公司 | HBase data migration method and device |
| CN110209653A (en) * | 2019-06-04 | 2019-09-06 | 中国农业银行股份有限公司 | HBase data migration method and moving apparatus |
| CN111241060A (en) * | 2020-01-08 | 2020-06-05 | 苏州科达科技股份有限公司 | Data migration method, system, device and storage medium |
| CN113377763A (en) * | 2020-03-10 | 2021-09-10 | 阿里巴巴集团控股有限公司 | Database table switching method and device, electronic equipment and computer storage medium |
| CN111459913B (en) * | 2020-03-31 | 2023-06-23 | 北京金山云网络技术有限公司 | Capacity expansion method and device of distributed database and electronic equipment |
| CN111459913A (en) * | 2020-03-31 | 2020-07-28 | 北京金山云网络技术有限公司 | Capacity expansion method and device of distributed database and electronic equipment |
| CN111538719A (en) * | 2020-04-30 | 2020-08-14 | 深圳前海微众银行股份有限公司 | Data migration method, device, equipment and computer storage medium |
| CN111538719B (en) * | 2020-04-30 | 2024-04-19 | 深圳前海微众银行股份有限公司 | Data migration method, device, equipment and computer storage medium |
| CN112069152A (en) * | 2020-09-08 | 2020-12-11 | 北京达佳互联信息技术有限公司 | Database cluster upgrading method, device, equipment and storage medium |
| CN112069152B (en) * | 2020-09-08 | 2023-10-03 | 北京达佳互联信息技术有限公司 | Database cluster upgrading method, device, equipment and storage medium |
| CN112463762A (en) * | 2020-11-06 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for cross-cluster real-time data migration and synchronization |
| CN112527767A (en) * | 2020-12-03 | 2021-03-19 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restart of distributed database |
| CN112527767B (en) * | 2020-12-03 | 2024-05-10 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restarting distributed database |
| CN112631994A (en) * | 2020-12-29 | 2021-04-09 | 深圳市商汤科技有限公司 | Data migration method and system |
| CN113032704A (en) * | 2021-02-24 | 2021-06-25 | 广州虎牙科技有限公司 | Data processing method, device, electronic equipment and medium |
| CN113032704B (en) * | 2021-02-24 | 2024-06-21 | 广州虎牙科技有限公司 | Data processing method, device, electronic equipment and medium |
| CN113438275A (en) * | 2021-05-27 | 2021-09-24 | 众安在线财产保险股份有限公司 | Data migration method and device, storage medium and data migration equipment |
| CN113438275B (en) * | 2021-05-27 | 2023-04-07 | 众安在线财产保险股份有限公司 | Data migration method and device, storage medium and data migration equipment |
| CN113742422A (en) * | 2021-08-20 | 2021-12-03 | 广州市易工品科技有限公司 | Data synchronization accuracy verification method and device |
| CN113946293A (en) * | 2021-10-27 | 2022-01-18 | 北京达佳互联信息技术有限公司 | Cluster data migration method and device, electronic equipment and storage medium |
| CN114546989B (en) * | 2022-02-22 | 2024-04-12 | 重庆长安汽车股份有限公司 | Hbase incremental data migration system, method and storage medium |
| CN114546989A (en) * | 2022-02-22 | 2022-05-27 | 重庆长安汽车股份有限公司 | Hbase incremental data migration system, method and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108628874B (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108628874A (en) | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of migrating data | |
| US10360536B2 (en) | Implementing a consistent ordering of operations in collaborative editing of shared content items | |
| US11985192B2 (en) | Synchronized content library | |
| KR101956236B1 (en) | Data replication technique in database management system | |
| CA2839014C (en) | Managing replicated virtual storage at recovery sites | |
| JP2022166013A (en) | Method, computer-readable medium and system for violation resolution in client synchronization | |
| JP6553822B2 (en) | Dividing and moving ranges in distributed systems | |
| US9460184B2 (en) | Application of a differential dataset to a data store using sequential change sets | |
| US9515878B2 (en) | Method, medium, and system for configuring a new node in a distributed memory network | |
| GB2564923A (en) | Managing digital assets stored as components and packaged files | |
| KR20200100173A (en) | Data replication and data failover within the database system | |
| AU2014274300A1 (en) | Access permissions for shared content | |
| US10809922B2 (en) | Providing data protection to destination storage objects on remote arrays in response to assignment of data protection to corresponding source storage objects on local arrays | |
| US11314719B2 (en) | Method for implementing change data capture in database management system | |
| CN103780638A (en) | Data synchronization method and system | |
| CN109863474A (en) | Update migratory system and method | |
| WO2022095366A1 (en) | Redis-based data reading method and apparatus, device, and readable storage medium | |
| CN108038153A (en) | The online data moving method and device of Hbase | |
| CN108446315A (en) | Big data moving method, device, equipment and storage medium | |
| CN102982171A (en) | Database synchronization method | |
| KR20190022600A (en) | Data replication technique in database management system | |
| CN107220248A (en) | A kind of method and apparatus for data storage | |
| CN109614383A (en) | Data copy method, device, electronic equipment and storage medium | |
| CN109254960B (en) | Method and device for migrating mass data of database | |
| CN103425550B (en) | A kind of system cloning process and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |