CN106294471A - Data Migration processing method and processing device - Google Patents
Data Migration processing method and processing device Download PDFInfo
- Publication number
- CN106294471A CN106294471A CN201510298975.1A CN201510298975A CN106294471A CN 106294471 A CN106294471 A CN 106294471A CN 201510298975 A CN201510298975 A CN 201510298975A CN 106294471 A CN106294471 A CN 106294471A
- Authority
- CN
- China
- Prior art keywords
- data
- nodal point
- primary nodal
- module
- secondary nodal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides a kind of Data Migration processing method and processing device, wherein, the method includes: receives and data will be specified to be migrated to the Data Migration request message of secondary nodal point by primary nodal point；Record is currently to the data manipulation modification information specifying data；By appointment data during primary nodal point migrates to secondary nodal point, according to data manipulation modification information to specify data carry out operation process.Being solved the problem needing to stop the read-write operation of data base in correlation technique during Data Migration and dilatation by the present invention, reduce in data migration process, database node stops the time window of service.
Description
Technical field
The present invention relates to the communications field, in particular to a kind of Data Migration processing method and processing device.
Background technology
The data of distributed data base system and table are distributed on each node, according to the data distribution rule of system, data
Can be distributed on different nodes according to different tables, it is also possible to be that the data of a table are distributed according to the Column Properties of table
On different nodes.When system data amount of storage increases, when original node cannot store, need system is deposited
Storage carries out dilatation, and the data that will be stored in a node are redistributed on two or more nodes.Another situation
Being after system runs a period of time, there is unbalanced situation in different node storages, in order to promote systematic function, needs
Data are migrated between node.
It is different from towards on-line analytical processing (Online Analytical Processing, referred to as OLAP) system, towards
Data are entered by Transaction Processing (On-Line Transaction Processing, referred to as OLTP) system the most simultaneously
Row reading and writing, update operation, and OLAP system is the most all repeatedly to read after write once.Therefore distributed data base
System must take into the impact of the read-write operation current on system during Data Migration and dilatation.Data migrate and
During dilatation, generally requiring consuming certain time, during this period of time system needs to stop the read-write operation of data base,
Or limiting the operation to data base, this certainly will have impact on the operation of business.
For in correlation technique, during Data Migration and dilatation, need to stop the problem of the read-write operation of data base,
Effective solution is not also proposed.
Summary of the invention
The invention provides a kind of Data Migration processing method and processing device, at least solving in correlation technique at Data Migration and
The problem needing during dilatation to stop the read-write operation of data base.
According to an aspect of the invention, it is provided a kind of Data Migration processing method, including: receive will specify data by
Primary nodal point migrates to the Data Migration request message of secondary nodal point；The current data manipulation to described appointment data of record becomes
More information；By described appointment data during described primary nodal point migrates to described secondary nodal point, according to described number
According to operation modification information, described appointment data are carried out operation process.
Further, described method also includes: from described primary nodal point, described appointment data are being migrated to described second section
After Dian, again described appointment data are carried out at operation in described secondary nodal point according to described data manipulation modification information
Reason；Stop that the data in the data in described primary nodal point and described secondary nodal point carry out operate change to process.
Further, stop carrying out operating at change to the data in the data in described primary nodal point and described secondary nodal point
Include after reason: node corresponding for described appointment data is updated to described secondary nodal point by described primary nodal point.
Further, wrap after node corresponding for described appointment data being updated to described secondary nodal point by described primary nodal point
Include: continue that the data in the data in described primary nodal point and described secondary nodal point are carried out operates change and process.
Further, described method also includes: from described primary nodal point, described appointment data are being migrated to described second section
After Dian, described appointment data are deleted from described primary nodal point.
Further, described data manipulation modification information includes the information of at least one of: update described appointment data
Information, revise described appointment data information, delete described appointment data information, increase data.
According to another aspect of the present invention, additionally providing a kind of Data Migration processing means, described device includes: receive
Module, will specify data to be migrated to the Data Migration request message of secondary nodal point by primary nodal point for receiving；Logging modle,
For recording the current data manipulation modification information to described appointment data；First processing module, for by described appointment
During data migrate to described secondary nodal point from described primary nodal point, according to described data manipulation modification information to described
Data are specified to carry out operation process.
Further, described device also includes: the second processing module, for by described appointment data from described first segment
Point migrate to described secondary nodal point after, according to described data manipulation modification information in described secondary nodal point again to described
Data are specified to carry out operation process；Stopping modular, for stopping the data in described primary nodal point and described secondary nodal point
In data carry out operate change process.
Further, described device also includes: more new module, for by node corresponding for described appointment data by described the
One node updates is described secondary nodal point.
Further, described device also includes: continue module, for continuing data in described primary nodal point and described
Data in secondary nodal point carry out operating change and process.
Further, described device also includes: removing module, for described appointment data being moved from described primary nodal point
After moving to described secondary nodal point, described appointment data are deleted from described primary nodal point.
Further, described data manipulation modification information includes the information of at least one of: update described appointment data
Information, revise described appointment data information, delete described appointment data information, increase data.
By the present invention, use and receive and data will be specified to be migrated to the Data Migration request message of secondary nodal point by primary nodal point；
Record is currently to the data manipulation modification information specifying data；Data will specified to migrate to secondary nodal point from primary nodal point
During, according to data manipulation modification information to specifying data to carry out operation process.Solve in correlation technique and move in data
The problem moving and need during dilatation to stop the read-write operation of data base, reduces in data migration process, data base
Node stops the time window of service.
Accompanying drawing explanation
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, the present invention
Schematic description and description be used for explaining the present invention, be not intended that inappropriate limitation of the present invention.In the accompanying drawings:
Fig. 1 is the flow chart of Data Migration processing method according to embodiments of the present invention；
Fig. 2 is the structured flowchart of Data Migration processing means according to embodiments of the present invention；
Fig. 3 is the structured flowchart () of Data Migration processing means according to embodiments of the present invention；
Fig. 4 is the structured flowchart (two) of Data Migration processing means according to embodiments of the present invention；
Fig. 5 is the structured flowchart (three) of Data Migration processing means according to embodiments of the present invention；
Fig. 6 is the structured flowchart (four) of Data Migration processing means according to embodiments of the present invention；
Fig. 7 is the system construction drawing of Data Migration according to embodiments of the present invention；
Fig. 8 is that the data of the Data Migration request that Data Migration module according to embodiments of the present invention is submitted to according to client are led
Go out to import process chart；
Fig. 9 is the stream that the monitoring in data migration process of the monitoring module DBMonitor according to embodiments of the present invention processes
Cheng Tu；
Figure 10 is the flow chart that the monitoring in data migration process of the routing module according to embodiments of the present invention processes.
Detailed description of the invention
Below with reference to accompanying drawing and describe the present invention in detail in conjunction with the embodiments.It should be noted that in the feelings do not conflicted
Under condition, the embodiment in the application and the feature in embodiment can be mutually combined.
Providing a kind of Data Migration processing method in the present embodiment, Fig. 1 is Data Migration according to embodiments of the present invention
The flow chart of processing method, as it is shown in figure 1, this flow process comprises the steps:
Step S102, receives and data will be specified to be migrated to the Data Migration request message of secondary nodal point by primary nodal point；
Step S104, record is currently to the data manipulation modification information specifying data；
Step S106, by appointment data during primary nodal point migrates to secondary nodal point, grasp according to above-mentioned data
Make modification information to specifying data to carry out operation process.
By above-mentioned steps, to the data manipulation modification information specifying data to operate in record primary nodal point, will refer to
During given data migrates to secondary nodal point from primary nodal point, still according to above-mentioned data manipulation modification information to specifying number
According to carrying out operation process, compared in correlation technique, data are migrating and during dilatation, and system needs to stop data
The read-write operation in storehouse, the consuming of the certain time caused, above-mentioned steps solves in correlation technique in Data Migration and dilatation
During need to stop the problem of read-write operation of data base, reduce in data migration process, database node stops
The time window of service.
By appointment data after primary nodal point migrates to secondary nodal point, in one alternate embodiment, according to above-mentioned number
In secondary nodal point, again appointment data are carried out operation process according to operation modification information, complete specifying data again
After operation processes, stop that the data in primary nodal point and the data in secondary nodal point are carried out operates change and process.So that
In specifying data after primary nodal point migrates to secondary nodal point, metadata information is updated.
During metadata information is updated, in one alternate embodiment, stopping the number in primary nodal point
After carrying out operating change process with the data in secondary nodal point, the node specifying data corresponding is updated by primary nodal point
For secondary nodal point.Hold so that data base manipulation statement being sent the most corresponding database node according to new metadata information
OK.
In one alternate embodiment, after the node specifying data corresponding is updated to secondary nodal point by primary nodal point, continue
Continuous carry out the data in primary nodal point and the data in secondary nodal point operates change process.
During metadata information is updated, in another alternative embodiment, data will specified from first segment
After point migrates to secondary nodal point, data will be specified to delete from primary nodal point.
Above-mentioned data manipulation modification information can comprise much information, is illustrated this below.An optional reality
Executing in example, above-mentioned data manipulation modification information may include that the information of these appointment data that updates, revises these appointment data
Information, delete these appointment data information, increase data.
Additionally providing a kind of Data Migration processing means in the present embodiment, this device is used for realizing above-described embodiment and preferably
Embodiment, had carried out repeating no more of explanation.As used below, term " module " can realize making a reservation for
The software of function and/or the combination of hardware.Although the device described by following example preferably realizes with software, but
It is hardware, or the realization of the combination of software and hardware also may and be contemplated.
Fig. 2 is the structured flowchart of Data Migration processing means according to embodiments of the present invention, as in figure 2 it is shown, this device bag
Include: receiver module 22, data will be specified to be migrated to the Data Migration request message of secondary nodal point by primary nodal point for receiving；
Logging modle 24, for recording currently to the data manipulation modification information specifying data；First processing module 26, is used for
By appointment data during primary nodal point migrates to secondary nodal point, according to this data manipulation modification information to specify number
According to carrying out operation process.
Fig. 3 is the structured flowchart () of Data Migration processing means according to embodiments of the present invention, as it is shown on figure 3, should
Device also includes: the second processing module 32, for by appointment data after primary nodal point migrates to secondary nodal point, root
In secondary nodal point, again appointment data are carried out operation process according to this data manipulation modification information；Stopping modular 34, is used for
Stop that the data in primary nodal point and the data in secondary nodal point carry out operate change to process.
Fig. 4 is the structured flowchart (two) of Data Migration processing means according to embodiments of the present invention, as shown in Figure 4, and should
Device also includes: more new module 44, for the node specifying data corresponding is updated to secondary nodal point by primary nodal point.
Fig. 5 is the structured flowchart (three) of Data Migration processing means according to embodiments of the present invention, as it is shown in figure 5, should
Device also includes: continue module 52, operates the data in primary nodal point and the data in secondary nodal point for continuation
Change processes.
Fig. 6 is the structured flowchart (four) of Data Migration processing means according to embodiments of the present invention, as shown in Figure 6, and should
Device also includes: removing module 62, for by appointments data after primary nodal point migrates to secondary nodal point, will appointment
Data are deleted from primary nodal point.
Alternatively, above-mentioned data manipulation modification information includes the information of at least one of: update these appointment data information,
The information revising these appointment data, the information deleting these appointment data, increase data.
It should be noted that above-mentioned modules can be by software or hardware realizes, for the latter, Ke Yitong
Cross in the following manner to realize, but be not limited to this: above-mentioned modules is respectively positioned in same processor；Or, each mould above-mentioned
Block lays respectively at first processor, the second processor and the 3rd processor ... in.
For the problems referred to above present in correlation technique, illustrate below in conjunction with concrete alternative embodiment.
This alternative embodiment proposes a kind of distributed data base Data Migration, dilatation and the method and system of redistribution.Should
Method, while Data Migration and dilatation, monitors the data manipulation at this node, and will need according to after data distributed update
The most together to be saved in the data of derivation in the data of new node write so that during data derive, it is not necessary to
Stop the read-write operation to data base.In data after new Node leading-in, at this moment stop the read-write of data base, complete be
The renewal of system metadata information and the cleaning of origin node data.By this system and method, data are being imported from origin node
During new node, greatly reduce system and stop the time of service, promote systematic function, improve availability.
In this alternative embodiment, after data migration server receives Data Migration request, notify former database node
Monitoring database monitoring instrument (Data Base Monitor, referred to as DBMonitor) starts to monitor all in former data
Data manipulation change on storehouse.Data migration server starts logically to operate, by data former database node simultaneously
Export to store in migration server.The monitoring DBMonitor monitoring all of data manipulation of meeting of data base, if former number
New database node is belonged to according to migration rules, then data operating record is arrived by DBMonitor according to the data of storehouse operation
In DBMonitor monitoring file.Data migration server, after data have derived, starts to will be stored in migration server
On data import to purpose database node.After in migration server, the data of storage all import,
DBMonitor plays back the data manipulation of record in monitoring file on new database node.When DBMonitor has played back
Bi Hou, migration server notice routing server stops the read-write operation of former data base's correlation table, and updates former data base joint
The data distribution metadata information of storage on point.After metadata information updates, migration server notice routing server
Carry out the route of data operation request according to new Data distribution information, and reply the read-write operation of former database node.Move
Moving server and begin a clean up the junk data of former database node, after all data scrubbings, now data are at former number
Migrating complete according to storehouse node and new database node, the metadata information on meta data server updates complete, so far simultaneously
Complete data migration operation.
Further, what the Data Migration of data migration server processed concretely comprises the following steps:
Data Migration module receives the Data Migration request of client.
Data Migration module notifies that the monitoring DBMonitor of former database node starts to monitor all of former database node
Data manipulation, including data renewal, revise, increase newly, the operation such as deletion.
DBMonitor is according to new routing rule, if the data of operation should occur on new database node, that
DBMonitor is by the monitoring file of data operating record to DBMonitor.
After the data of former database node are derived by Data Migration module, Data Migration module stops former data base joint
The reading of point, starts to import data toward new database node.Now DBMonitor continues to monitor former database node
Data manipulation.
After in Data Migration module, all data import, Data Migration module notice DBMonitor is in new data
Playback of data on the node of storehouse.So-called playback of data, the data change operation that also will perform on former data base before exists again
New database node performs once.
After DBMonitor receives playback request, start the data manipulation of record in monitoring file one by one in new data
Operate on node, including new database node data, revise, increase newly, deletion action.
DBMonitor, during playback, still can monitor the operation of former database node.When DBMoitor owns
Data readback after, DBMonitor notice Data Migration module data playback complete.
After Data Migration receives the complete message of playback of DBMonitor, DBMonitor notice routing module stops former number
According to storehouse and the read-write operation of the data table related of new database node.
Data Migration module notice metadatabase server updates metadata information according to new Data distribution information.
After metadata information updates, Data Migration module notice routing server recovers the read-write operation of former data base.
Data Migration module begins a clean up the junk data of former database node.
After Data Migration module completes the garbage data cleaning of database node, notice client completes data migration operation.
Below in conjunction with the accompanying drawings the present invention " a kind of distributed data base Data Migration and redistribution method and system " is entered in detail
Explanation.
Fig. 7 is the system construction drawing of Data Migration according to embodiments of the present invention, as it is shown in fig. 7, be the system of the present invention
Structure chart.System structure specifically includes that client 101, routing module 102 (perform the merit of above-mentioned stopping modular 34
Can), metadata cache module 103, Data Migration module 104 (perform above-mentioned receiver module the 22, first processing module
26 and second function of processing module 32), the unit data base 105 of distributed experiment & measurement system, database manipulation monitoring
Module 106 (performing the function of above-mentioned logging modle 24).
Client 101: client modules, user is carried by client 101 (including computer, mobile phone, browser etc.)
Hand over transaction structure query language (Structured Query Language, referred to as SQL) statement etc..
Routing module 102: routing module, is will to perform the SQL statement that module issues, select suitable data base to connect
Issue SQL statement to go to perform.Routing module can do the process of SQL statement for distributed SQL statement, as in data
Migration phase, routing module needs SQL statement is processed into the SQL carrying routing iinformation, all can increase as additions and deletions change to look into
Add routing iinformation.
Metadata cache module 103: metadata cache module, for the caching of metadata, including table definition information, distribution
Formula data base each Node distribution information.Metadata cache module can carry out renewal and the inquiry operation of metadata.Work as system
During metadata information change, the information that meeting synchronized update metadata cache module preserves.Meanwhile, metadata cache module connects
Receive the inquiry operation of other modules, return metadata information.
Data Migration module 104: Data Migration module, for the migration of data, including the importing and exporting of data, system
The related work of the Data Migration such as dilatation, fast resampling.Data Migration module can receive the file that database node is derived,
And the file of derivation is imported to new database node again.The centre that Data Migration module can derive with document form storage
File.
The unit data base 105 of distributed experiment & measurement system: the unit data base of distributed experiment & measurement system, can receive also
Perform the database language of distributed experiment & measurement system.Wherein database language includes DDL, data manipulation language
Speech, data query language, DCL, transaction controlling language etc..
Database manipulation monitoring module 106: database manipulation monitoring module DBMonitor, for monitoring distributed data
Each database positioning of storehouse cluster, he can be by the language of data base's all modifications database data with binary form record
In journal file, such as insert, update, delete etc..DBMonitor can record the change of all records, will be every
The amendment record of data line all recorded in journal file.Journal file does not include database language, for inquiry language
Sentences etc., because being not related to the change of data, also will not recorded in journal file.
Fig. 8 is that the data of the Data Migration request that Data Migration module according to embodiments of the present invention is submitted to according to client are led
Go out to import process chart.After Data Migration module receives Data Migration request, start to notify that DBMonitor monitoring is former
Data manipulation on database node, starts afterwards to export in file data, then imports to new by the file of derivation
In database node.After data import, then the data manipulation daily record monitored by DBMonitor is again purpose data base
Node performs.Import the read-write operation that the complete routing module of notice afterwards stops the correlation table of metadatabase, and more Singapore dollar number
It is believed that breath.Update complete after, Data Migration module notice routing module recover former data base's correlation table read-write operation power
Limit, and begin a clean up the junk data of former database node.After treating garbage data cleaning, Data Migration module notifies
Customer data migrates complete, and operating result is returned to client.As shown in Figure 8, this flow process specifically includes following steps:
Step S801: Data Migration module receives the Data Migration request that client issues.
Step S802: after Data Migration module receives the Data Migration request of client, notice DBMonitor monitoring
Data manipulation.Wherein the details of DBMonitor monitoring database manipulation are shown in that Fig. 9 describes.
Step S803: Data Migration module starts to export in file the data of former data base.
Step S804: judge whether the data of former database node derive complete, if it is not, continue to derive, weight
Multiple step S803, otherwise enters step S805.
Step S805: the file of derivation is imported to new database node by Data Migration module.
Step S806: it is complete that Data Migration module judges whether file imports, if data do not import complete, then weighs
Multiple step S805, otherwise enters step S807.
Step S807: Data Migration module notice DBMonitor by the data readback in journal file to purpose data base
Node.So-called playback of data, also will exist in journal file in the data performed on the former data base change operation of record again
New database node performs once.It should be noted that, during playback, DBMonitor need nonetheless remain for monitoring
The data manipulation of former data base.
Step S808: it is complete, without playback that Data Migration module judges whether DBMonitor journal file plays back
Complete, then continue to repeat step S807, otherwise enter step S809.
Step S809: Data Migration module gets data file and imports complete, the daily record literary composition of DBMonitor monitoring simultaneously
Part has played back complete the most, then Data Migration module notice routing module stops the read-write operation of Relational database table.Wherein
To the Read-write Catrol of former database node can be table, storehouse granularity on carry out access limit control.Such as a big table,
Data need to carry out cutting and by a part of Data Migration to new data base according to distribution key, then now routing module
It is accomplished by stopping former database node, the read-write operation of new database node correlation table.
Step S810: Data Migration module notice metadata cache module updates metadata information.After doing Data Migration,
The memory node information of data will change, and metadata information needs the distributed intelligence of renewal table.Routing module is receiving
To database manipulation ask time, can query metadata cache information, if relate to the operation of former data base's correlation table,
The most just according to the distributed intelligence of new table, database manipulation request should be routed to new database node operation.
Step S811: Data Migration module is after completing metadata updates, and notice routing module recovers Relational database joint
The operating right of point.
Step S812: Data Migration module begins a clean up the junk data of former database node, is wherein clearing up former data
During the junk data of storehouse node, routing module needs to do the SQL statement mailing to former database node to control, and specifically flows
Journey is as shown in Figure 10.
Step S813: judge whether junk data cleans out, if do not cleaned out, then enters step S812 and continues
Cleaning junk data；If garbage data cleaning is complete, then enter step S814.
Step S814: notice client garbage data cleaning is complete, completes Data Migration work.Data Migration module is led to
Know that routing module stops monitoring the action statement of the correlation table of former database node.
Fig. 9 is the stream that the monitoring in data migration process of the monitoring module DBMonitor according to embodiments of the present invention processes
Cheng Tu.As it is shown on figure 3, be the flow chart of monitoring module DBMonitor monitoring process in data migration process, tool
Body step is as described below:
Step S901: monitoring module DBMonitor starts to monitor the data manipulation of former database node.He can be by data
The language of storehouse all modifications database data recorded in journal file with binary form, as inserted (insert), more
Newly (update), deletion (delete) etc..DBMonitor can record the change of all records, by every data line
Amendment record all recorded in journal file.Journal file does not include database language, for query statement etc., because of not
Relate to the change of data, also will not recorded in journal file.
Step S902, DBMonitor judges the data manipulation monitored, it is judged that database language operates the need of newly
Database node perform.If need not, enter step S904, otherwise enter step S903.
Step S903, is saved in the data record that database language operates in journal file.Recorded in journal file
Content comprise this data storehouse operation type, and this operation before and after data.As for update, record
Action type is for inserting, and the data of insertion is recorded；For updating operation, record action type is renewal,
And will update before and after data record.Similarly, relate to the operation of data change, all action type is recorded
Come, and record data.For being not related to the operation of data change, and it is relevant to be not related to new database node
The change of table, all need not recorded monitoring journal file.
Step S904, it may be judged whether stop monitoring data manipulation, if do not stopped, continuing monitoring, entering step S901；
Otherwise complete data manipulation monitoring.
Figure 10 is the flow chart that the monitoring in data migration process of the routing module according to embodiments of the present invention processes, as
Shown in Figure 10, the flow chart that routing module monitoring in data migration process processes.Routing module is in data metadata
Before changing, directly according to original metadata information, data base manipulation statement is mail to corresponding database node perform.
After metadata information change, need, according to new metadata information, data base manipulation statement is dealt into correspondence database node and hold
OK.In order to promote database performance and availability in transition process, when doing Data Migration, original database node is still
So need to receive the operation read and write.The most original database node remains new database node needs the complete of migration
Portion's data, and the data write in transition process.Data Migration is to after new database node, and original data base saves
Point data be accomplished by doing cleaning work, need in transition process to mail to original database manipulation language need carry out
Processing, Figure 10 flow chart describes detailed process, and its step is as follows:
Step S1001, routing module receives database manipulation language.
Step S1002, routing module goes to read its metadata information according to the table that database manipulation language relates to.Unit's number
It is believed that breath includes the distribution situation of table, such as the scope according to major key, determine that data are to fall on corresponding database node.
Step S1003, routing module is according to the metadata information read, it is judged that whether database manipulation relates to migrate
Data.Without relate to migrate data, then enter step S1005；Otherwise, step S1004 is entered.
Step S1004, if database manipulation data relate to new migration data, then be accomplished by its SQL statement
Process, be revised as database statement only performing in the scope of data of former data base.As one updates statement, more
New scope of data contains the data of new and old two database nodes.Routing module is by the operation road to new database node
Performed by its database node, legacy data storehouse node-routing is performed to former database node.But due to former
Database node contains the junk data not cleaned out, and the operation being routed to former database node may operating portion
Divide junk data, it is therefore desirable to the scope of data this operated is the due scope of former database node after being limited in Data Migration.
Step S1005, data base manipulation statement is issued to the upper execution of data base by routing module.
Step S1006, routing module judges that data scrubbing is the most complete, if the most complete, then enter step S1001 and continues
Continuous monitoring, otherwise terminates the Route monitoring of database manipulation language.
In sum, during carrying out Data Migration, monitored the reading of new data by migration server of the present invention simultaneously
Take and write operation, the data manipulation that record will be carried out on new node.By the time after data manipulation, more Singapore dollar number
When it is believed that breath, notice routing module stops the read-write operation of former database node.Need not during data exporting
Stop the read-write operation of former database node, updating metadatabase information when, only stop read-write operation.So system
In carrying out data migration process, do not interfere with the normal read-write operation of system, thus realize the reality of distributed data base system
Time smooth linear dilatation, improve systematic function, improve availability.Owing to the renewal of metadata information can complete immediately,
Therefore the time window that database node stops servicing is minimized, and reduces the data migration process impact on system service.
The present invention is applicable to the occasion that distributed relation database needs to carry out the logic migration operation of data, as Data Migration,
Data backup, data-base capacity-enlarging etc. operate.
In another embodiment, additionally providing a kind of software, this software is used for performing above-described embodiment and being preferable to carry out
Technical scheme described in mode.
In another embodiment, additionally providing a kind of storage medium, in this storage medium, storage has above-mentioned software, should
Storage medium includes but not limited to: CD, floppy disk, hard disk, scratch pad memory etc..
Obviously, those skilled in the art should be understood that each module of the above-mentioned present invention or each step can be with general
Calculating device to realize, they can concentrate on single calculating device, or be distributed in multiple calculating device and formed
Network on, alternatively, they can realize, it is thus possible to by them with calculating the executable program code of device
Storage is performed by calculating device in the storage device, and in some cases, can hold with the order being different from herein
Step shown or described by row, or they are fabricated to respectively each integrated circuit modules, or by many in them
Individual module or step are fabricated to single integrated circuit module and realize.So, the present invention is not restricted to any specific hardware
Combine with software.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for the technology of this area
For personnel, the present invention can have various modifications and variations.All within the spirit and principles in the present invention, that is made is any
Amendment, equivalent, improvement etc., should be included within the scope of the present invention.
Claims (12)
1. a Data Migration processing method, it is characterised in that including:
Receive and data will be specified to be migrated to the Data Migration request message of secondary nodal point by primary nodal point；
The current data manipulation modification information to described appointment data of record；
By described appointment data during described primary nodal point migrates to described secondary nodal point, according to described number
According to operation modification information, described appointment data are carried out operation process.
Method the most according to claim 1, it is characterised in that described method also includes:
By described appointment data after described primary nodal point migrates to described secondary nodal point, grasp according to described data
Make modification information and again described appointment data are carried out operation process in described secondary nodal point；
Stop that the data in the data in described primary nodal point and described secondary nodal point carry out operate change to process.
Method the most according to claim 2, it is characterised in that stop the data in described primary nodal point and described the
Data in two nodes include after carrying out operating change process:
Node corresponding for described appointment data is updated to described secondary nodal point by described primary nodal point.
Method the most according to claim 3, it is characterised in that by node corresponding for described appointment data by described first
Node updates is to include after described secondary nodal point:
Continue that the data in the data in described primary nodal point and described secondary nodal point carry out operate change to process.
Method the most according to claim 1, it is characterised in that described method also includes:
By described appointment data after described primary nodal point migrates to described secondary nodal point, by described appointment data
Delete from described primary nodal point.
Method the most according to any one of claim 1 to 5, it is characterised in that described data manipulation modification information bag
Include the information of at least one of:
The information updating described appointment data, the information revising described appointment data, delete the letter of described appointment data
Breath, increase data.
7. a Data Migration processing means, it is characterised in that described device includes:
Receiver module, disappears the Data Migration request specifying data to be migrated to secondary nodal point by primary nodal point for receiving
Breath；
Logging modle, for recording the current data manipulation modification information to described appointment data；
First processing module, for migrating to described secondary nodal point by described appointment data from described primary nodal point
During, according to described data manipulation modification information, described appointment data are carried out operation process.
Device the most according to claim 7, it is characterised in that described device also includes:
Second processing module, for by described appointment data from described primary nodal point migrate to described secondary nodal point it
After, again described appointment data are carried out at operation in described secondary nodal point according to described data manipulation modification information
Reason；
Data in data in described primary nodal point and described secondary nodal point are grasped by stopping modular for stopping
Make change to process.
Device the most according to claim 8, it is characterised in that described device also includes:
More new module, for being updated to described second section by node corresponding for described appointment data by described primary nodal point
Point.
Device the most according to claim 9, it is characterised in that described device also includes:
Continue module, for continuation, the data in the data in described primary nodal point and described secondary nodal point are grasped
Make change to process.
11. devices according to claim 7, it is characterised in that described device also includes:
Removing module, for by described appointment data after described primary nodal point migrates to described secondary nodal point,
Described appointment data are deleted from described primary nodal point.
12. according to the device according to any one of claim 7 to 11, it is characterised in that described data manipulation modification information
Information including at least one of: the information updating described appointment data, the information revising described appointment data,
Delete the information of described appointment data, increase data.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510298975.1A CN106294471A (en) | 2015-06-03 | 2015-06-03 | Data Migration processing method and processing device |
| PCT/CN2016/080493 WO2016192496A1 (en) | 2015-06-03 | 2016-04-28 | Data migration processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510298975.1A CN106294471A (en) | 2015-06-03 | 2015-06-03 | Data Migration processing method and processing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106294471A true CN106294471A (en) | 2017-01-04 |
Family
ID=57440177
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510298975.1A Withdrawn CN106294471A (en) | 2015-06-03 | 2015-06-03 | Data Migration processing method and processing device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106294471A (en) |
| WO (1) | WO2016192496A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN108509150A (en) * | 2018-03-08 | 2018-09-07 | 华为技术有限公司 | Data processing method and device |
| CN108763247A (en) * | 2018-03-29 | 2018-11-06 | 阿里巴巴集团控股有限公司 | The method and device of user's request is handled in data migration process |
| CN108875042A (en) * | 2018-06-27 | 2018-11-23 | 中国农业银行股份有限公司 | A kind of mixing on-line analysing processing system and data query method |
| CN108932282A (en) * | 2018-05-18 | 2018-12-04 | 腾讯科技（深圳）有限公司 | A kind of database migration method, apparatus and storage medium |
| CN111858469A (en) * | 2020-07-24 | 2020-10-30 | 成都成信高科信息技术有限公司 | Self-adaptive hierarchical storage method based on time sliding window |
| CN113407462A (en) * | 2021-06-16 | 2021-09-17 | 新华三信息安全技术有限公司 | Data processing method and device, electronic equipment and medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111723148B (en) * | 2020-04-10 | 2024-01-26 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102103629A (en) * | 2010-12-14 | 2011-06-22 | 西北工业大学 | Online data migration method |
| CN102142027A (en) * | 2011-02-23 | 2011-08-03 | 南京智尚丰软件有限公司 | Adaptive method for data integration |
| US8140631B2 (en) * | 2004-05-14 | 2012-03-20 | Mirapoint Software, Inc. | Method for mailbox migration |
| US20120254175A1 (en) * | 2011-04-01 | 2012-10-04 | Eliot Horowitz | System and method for optimizing data migration in a partitioned database |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN103164167A (en) * | 2011-12-15 | 2013-06-19 | 深圳市腾讯计算机系统有限公司 | Data migration method and data migration device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500146B (en) * | 2013-09-30 | 2016-04-27 | 北京邮电大学 | Magnetic disk of virtual machine stores data migration method and system |
-
2015
- 2015-06-03 CN CN201510298975.1A patent/CN106294471A/en not_active Withdrawn
-
2016
- 2016-04-28 WO PCT/CN2016/080493 patent/WO2016192496A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8140631B2 (en) * | 2004-05-14 | 2012-03-20 | Mirapoint Software, Inc. | Method for mailbox migration |
| CN102103629A (en) * | 2010-12-14 | 2011-06-22 | 西北工业大学 | Online data migration method |
| CN102142027A (en) * | 2011-02-23 | 2011-08-03 | 南京智尚丰软件有限公司 | Adaptive method for data integration |
| US20120254175A1 (en) * | 2011-04-01 | 2012-10-04 | Eliot Horowitz | System and method for optimizing data migration in a partitioned database |
| CN103164167A (en) * | 2011-12-15 | 2013-06-19 | 深圳市腾讯计算机系统有限公司 | Data migration method and data migration device |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107357896A (en) * | 2017-07-13 | 2017-11-17 | 北京小度信息科技有限公司 | Expansion method, device, system and the data base cluster system of data-base cluster |
| CN108509150A (en) * | 2018-03-08 | 2018-09-07 | 华为技术有限公司 | Data processing method and device |
| CN108763247A (en) * | 2018-03-29 | 2018-11-06 | 阿里巴巴集团控股有限公司 | The method and device of user's request is handled in data migration process |
| CN108763247B (en) * | 2018-03-29 | 2021-12-07 | 创新先进技术有限公司 | Method and device for processing user request in data migration process |
| CN108932282A (en) * | 2018-05-18 | 2018-12-04 | 腾讯科技（深圳）有限公司 | A kind of database migration method, apparatus and storage medium |
| CN108875042A (en) * | 2018-06-27 | 2018-11-23 | 中国农业银行股份有限公司 | A kind of mixing on-line analysing processing system and data query method |
| CN111858469A (en) * | 2020-07-24 | 2020-10-30 | 成都成信高科信息技术有限公司 | Self-adaptive hierarchical storage method based on time sliding window |
| CN111858469B (en) * | 2020-07-24 | 2024-01-26 | 成都成信高科信息技术有限公司 | Self-adaptive hierarchical storage method based on time sliding window |
| CN113407462A (en) * | 2021-06-16 | 2021-09-17 | 新华三信息安全技术有限公司 | Data processing method and device, electronic equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2016192496A1 (en) | 2016-12-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106294471A (en) | Data Migration processing method and processing device | |
| CN101278540B (en) | System and method to maintain coherence of cache contents in a multi-tier software system aimed at interfacing large databases | |
| US7680846B2 (en) | File management program, file management apparatus and file management method | |
| Santos et al. | Real-time data warehouse loading methodology | |
| CN105930387A (en) | Data operation system and method based on data routing and sharding | |
| JP2001313639A (en) | System and method for managing network configuration data and recording medium | |
| CN103793493B (en) | A kind of method and system for handling car-mounted terminal mass data | |
| CN108369601A (en) | Promotion attribute in relational structure data | |
| CN101523391A (en) | Information search system, information search method, and program | |
| CN108369599A (en) | Duplication control between redundant data center | |
| CN101131697A (en) | System and method for moving records between partitions | |
| KR102038529B1 (en) | System for processing real-time data modification of in-memory database | |
| CN110019469A (en) | Distributed data base data processing method, device, storage medium and electronic device | |
| CN102779088B (en) | Transaction facility, transaction methods and transaction processing program | |
| Santos et al. | Optimizing data warehouse loading procedures for enabling useful-time data warehousing | |
| JP5412995B2 (en) | Database control apparatus, database control method, and database control program in program model checking | |
| CN111930716A (en) | Database capacity expansion method, device and system | |
| CN103365987A (en) | Clustered database system and data processing method based on shared-disk framework | |
| KR101955376B1 (en) | Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method | |
| US20060129521A1 (en) | System and method for restoring a file directory structure | |
| JP5907251B2 (en) | Database management method, program, and information processing apparatus | |
| WO2016117007A1 (en) | Database system and database management method | |
| CN109359100A (en) | The visual modeling method of PB grades of historical datas and online data calculated in real time | |
| US7043491B1 (en) | Partition exchange technique for operating a data warehousing system | |
| CN114385577A (en) | Distributed file system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| CB03 | Change of inventor or designer information |
Inventor after: Dai Lijian Inventor after: Liu Wentao Inventor after: Deng Qiong Inventor after: Chen Jianfeng Inventor before: Dai Lijian |
|
| CB03 | Change of inventor or designer information | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180125 Address after: 518057 Nanshan District science and technology, Guangdong Province, South Road, No. 55, No. Applicant after: ZTE Corporation Applicant after: China Citic Bank Co., Ltd. Address before: 518057 Nanshan District science and technology, Guangdong Province, South Road, No. 55, No. Applicant before: ZTE Corporation |
|
| TA01 | Transfer of patent application right | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20170104 |
|
| WW01 | Invention patent application withdrawn after publication |