CN108280148A - A kind of data migration method and data migration server - Google Patents
A kind of data migration method and data migration server Download PDFInfo
- Publication number
- CN108280148A CN108280148A CN201810002213.6A CN201810002213A CN108280148A CN 108280148 A CN108280148 A CN 108280148A CN 201810002213 A CN201810002213 A CN 201810002213A CN 108280148 A CN108280148 A CN 108280148A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- target database
- written
- migrated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a kind of data migration method and data migration servers.The data migration method includes：Receive data migration task set；The data migration task set carries source database mark, multiple target databases mark and Data distribution information；According to the Data distribution information, target database mark to be written is determined in the multiple target database mark；It is identified from the source database and reads data to be migrated in corresponding source database, and the data write-in target database to be written to be migrated is identified in corresponding target database.According to embodiments of the present invention, while realizing that data point library divides table to migrate, without the migration rules of setting large amount of complex, reduce the development and maintenance work of user, reduce the cost of Data Migration.
Description
Technical field
Field of data migration of the embodiment of the present invention, especially a kind of data migration method, and, a kind of Data Migration service
Device.
Background technology
Due to the rise of internet industry, the rise and fast development of distributed computing technology are driven, more and more tradition
Industry carries out data storage using distributed system architecture.In distributed system architecture, a most basic technology, is exactly logarithm
According to horizontal fractionation is carried out, i.e. table technology is divided in a point library.Table technology is divided in so-called point of library, is exactly to pass through the data in a tables of data
Certain rule is split in different databases or different tables of data.
It when table is divided in realization point library, needs to migrate a large amount of data, especially the historical data of some cores
Migration.Currently, be typically realize Data Migrations using some existing data extraction tools, such as Kettle, Datastage and
The data processing softwares such as Informatica.
However, when carrying out Data Migration using above-mentioned data processing software, because the data of source database are write
In the tables of data for entering at most a database, therefore, it is necessary to which the migration rules of large amount of complex are arranged, cause to carry out to need when Data Migration
A large amount of development and maintenance work are carried out, the cost of Data Migration is increased.
Therefore, there is the higher problems of Data Migration cost for the data migration method of the prior art.
Invention content
The present invention provides a kind of data migration methods, and, a kind of data migration server, to solve the prior art
There is the higher problems of Data Migration cost for data migration method.
In order to solve the above-mentioned technical problem, it the embodiment of the invention discloses a kind of data migration method, is moved applied to data
Server is moved, the method includes：
Receive data migration task set；The data migration task set carries source database mark, multiple number of targets
According to library mark and Data distribution information；
According to the Data distribution information, target database mark to be written is determined in the multiple target database mark
Know；
It is identified in corresponding source database from the source database and reads data to be migrated, and the data to be migrated are write
Enter the target database to be written to identify in corresponding target database.
Optionally, the method further includes：
When the corresponding migrating data amount of the data migration task set is less than predetermined threshold value, for the Data Migration
The single processing thread of set of tasks configuration；
Described identified from the source database reads data to be migrated in corresponding source database, and by the number to be migrated
The step in corresponding target database is identified according to the target database to be written is written, including：
It calls the single processing thread to be identified in corresponding source database from the source database and reads data to be migrated,
And the data write-in target database to be written to be migrated is identified in corresponding target database.
Optionally, the data migration task set further includes with corresponding data distribution characteristics, the method：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the Data Migration
When the corresponding data distribution characteristics of set of tasks meet default distribution characteristics, configured for the data migration task set single
Thread is handled as reading thread, and, it is queue assignment thread to configure single processing thread, and, configure at least one place
Lineation journey is write-in thread；
Described identified from the source database reads data to be migrated in corresponding source database, and by the number to be migrated
The step in corresponding target database is identified according to the target database to be written is written, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, with
And call the queue assignment thread that the data to be migrated of reading are formed multiple data queues, call said write thread will
Data to be migrated in the multiple data queue are sequentially written in the target database to be written and identify corresponding target data
In library.
Optionally, the method further includes：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the Data Migration
When the corresponding data distribution characteristics of set of tasks do not meet default distribution characteristics, configured for the data migration task set single
A processing thread, which is used as, reads thread, and, it is write-in thread to configure multiple processing threads；
Described identified from the source database reads data to be migrated in corresponding source database, and by the number to be migrated
The step in corresponding target database is identified according to the target database to be written is written, including：
It calls the reading thread to identify from the source database to read data to be migrated in corresponding source database and incite somebody to action
The data to be migrated read form individual data queue, and, call multiple write-in threads will be in the individual data queue
The data write-in target database to be written to be migrated identifies in corresponding target database.
Optionally, the data migration task set further includes with corresponding data manipulation amount, the method：
When the corresponding data manipulation amount of the data migration task set is more than predetermined threshold value, for the Data Migration
The single processing thread of set of tasks configuration, which is used as, reads thread, and, it is queue assignment thread to configure multiple processing threads, with
And at least one processing thread of configuration is write-in thread；
Described identified from the source database reads data to be migrated in corresponding source database, and by the number to be migrated
The step in corresponding target database is identified according to the target database to be written is written, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, with
And call multiple queue assignment threads that the data to be migrated of reading are formed multiple data queues, call said write thread will
Data to be migrated in the multiple data queue are sequentially written in the target database to be written and identify corresponding target data
In library.
Optionally, the data to be migrated include multiple subdatas, described that the data write-in to be migrated is described to be written
Enter target database and identifies step in corresponding target database, including：
The multiple subdata batch write-in to the target database to be written is identified into corresponding target database.
Optionally, described that the write-in of the multiple subdata batch is corresponding to the target database mark to be written
After the step of target database, the method further includes：
When receiving the write-in notice extremely that the target database is sent, extracted at least from the multiple subdata
One subdata；
At least one subdata is written to the target database to be written and identifies corresponding target database, and
Judge whether at least one subdata is successfully written；
If so, the step of extracting at least one subdata from the multiple subdata described in returning；
It is unsuccessfully identified if it is not, being then directed at least one subdata addition write-in.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the method further includes：
When receiving data breakpoint migration task, the maximum data ident value in the multiple tables of data is searched, as
Target data break value；
Breakpoint transmission migration process is executed according to the target data break value.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the method further includes：
When receiving data breakpoint migration task, its maximum data mark is searched respectively in the multiple tables of data
Value, as multiple candidate data break values；
In the multiple candidate data break value, minimum candidate data break value is searched, as target data break value；
Breakpoint transmission migration process is executed according to the target data break value.
In order to solve the above-mentioned technical problem, the embodiment of the invention also discloses a kind of data migration server, the data
Migration server includes：
Task receiving module, for receiving data migration task set；The data migration task set carries source data
Library mark, multiple target databases mark and Data distribution information；
Determining module is identified, for according to the Data distribution information, being determined in the multiple target database mark
Target database mark to be written；
Data Migration module reads data to be migrated for being identified in corresponding source database from the source database, and
The data write-in target database to be written to be migrated is identified in corresponding target database.
According to embodiments of the present invention, by setting Data distribution information, and migrating data is determined according to Data distribution information
Distributing position in multiple target databases and multiple tables of data, by most a database of the Data Migration of source database
Tables of data when, while realizing that data point libraries divides table to migrate, without the migration rules of setting large amount of complex, reduce user
Development and maintenance work, reduce the cost of Data Migration.
Description of the drawings
Fig. 1 is the flow chart for the data migration method that the embodiment of the present invention one provides；
Fig. 2 is the structural schematic diagram of the data migration server cluster of the embodiment of the present invention；
Fig. 3 is the data flow schematic diagram of a data migration example of the embodiment of the present invention；
Fig. 4 is the data flow schematic diagram of another Data Migration example of the embodiment of the present invention；
Fig. 5 is the flow chart of data migration method provided by Embodiment 2 of the present invention；
Fig. 6 is the schematic diagram of the parallel migration strategy of the embodiment of the present invention；
Fig. 7 is the schematic diagram of the asynchronous Fair migration strategies of the embodiment of the present invention；
Fig. 8 is the schematic diagram of the asynchronous Crazy migration strategies of the embodiment of the present invention；
Fig. 9 is the schematic diagram of the asynchronous Crazy migration strategies of the embodiment of the present invention；
Figure 10 is a kind of schematic diagram of breakpoint moving method of the embodiment of the present invention；
Figure 11 is the schematic diagram of another breakpoint moving method of the embodiment of the present invention；
Figure 12 is the structure diagram for the data migration server that the embodiment of the present invention three provides；
Figure 13 is the structure diagram for the data migration server that the embodiment of the present invention four provides.
Specific implementation mode
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
Embodiment one
Fig. 1 is the flow chart for the data migration method that the embodiment of the present invention one provides, and this method takes applied to Data Migration
Business device, the data migration server are configured with multiple processing threads, and this method can specifically include following steps：
Step 110, data migration task set is received；The data migration task set carries source database and identifies, is more
A target database mark and Data distribution information.
Firstly the need of explanation, a data migration server can be set, sheet is executed by the data migration server
The data migration method of inventive embodiments.In practical applications, data migration server can have multiple, by multiple Data Migrations
Server forms a data migration server cluster.
Fig. 2 is the structural schematic diagram of the data migration server cluster of the embodiment of the present invention.It can be seen that Data Migration
Server cluster may include multiple data migration servers, each data migration server can respectively at least one source number
Data interaction is carried out according to library and at least one target database.It, can when by the Data Migration of source database to target database
According to specific data migration task, data are read from wherein one or more source databases, then by the data of reading batch
In amount write-in to wherein one or more target databases.In addition, being directed to data migration server cluster, one can also be arranged
Resources bank is migrated, the system configuration for configuring each data migration server such as migrates set of tasks information, execution journal
Deng.
In the concrete realization, user can submit a Data Migration operation to data migration server.Wherein, a number
Can include a multiple data migration tasks according to migration operation, to form a Data Migration set of tasks.Data as a result,
Migration server can receive data migration task set.
Each data migration task in data migration task set can carry user-defined source database mark
Know, target database mark and Data distribution information.Wherein, source database mark has corresponding source database, target data
Library mark has corresponding target database.
In addition, the data in database, which are the modes based on tables of data, carries out classification storage, in each database, can have
There is one or more tables of data.Therefore, each Data Migration operation can also include data in addition to the mark including database
The mark of table, for the storage location where location data.
Wherein, Data distribution information for indicate how divide source database in some tables of data multiple data, and,
Some tables of data in each data target database of being written after division.
Step 120, according to the Data distribution information, target to be written is determined in the multiple target database mark
Database Identification.
In the specific implementation, according to Data distribution information, it may be determined that a target database mark.For example, current receive
Data migration task set in, the target database mark that one of data migration task carries includes ds1, ds2, ds3
And ds4, wherein target database ds1, ds2, ds3 and ds4 are respectively provided with 2 tables of data.Data distribution information be [2,2,2,
2], indicate currently to need to migrate multiple data of source database into 8 tables of data of 4 target database, moreover, working as
It is preceding that 8 tables of data being written is needed to be evenly distributed in 4 different target databases to be written.Hence, it can be determined that target
Database Identification includes that ds1, ds2, ds3 and ds4 are that above-mentioned target database to be written identifies.
Step 130, it is identified in corresponding source database from the source database and reads data to be migrated, and wait moving by described
The data write-in target database to be written is moved to identify in corresponding target database.
In the specific implementation, one or more processing threads of data migration server can be called, it will be in source database
Data to be migrated are read by processing thread and are written into the corresponding target database of target database to be written mark.
When calling processing thread to carry out Data Migration, a variety of different migration strategies may be used.
For example, simultaneously and concurrently migration strategy, that is, for the single processing thread of data migration task set configuration, by the list
A entire data migration task set of processing thread process, this individually handles thread and is responsible for reading data from source database, simultaneously
It is responsible for the data of reading being synchronously written target database, realizes the synchronous migration to data.If there is multiple Data Migrations are appointed
Business set, then respectively by multiple single parallel migrations for handling the multiple data migration task set of thread process, realizing to data.
In another example asynchronous Fair migration strategies, that is, multiple processing threads are configured for data migration task set, by list
Thread is read in a processing thread conduct, and data are read from source database, by individually handling thread as queue assignment thread, is incited somebody to action
The data of reading are distributed into a certain data queue, will be corresponding with write-in thread by multiple processing threads as write-in thread
Data in data queue, are sequentially written in target database.
In another example asynchronous Crazy migration strategies, that is, multiple processing threads are configured for data migration task set, by list
A processing thread reads data, and the data of reading are formed individual data queue as thread is read from source database, by
Multiple processing threads, will be in the data write-in target database in data queue as write-in thread.
In another example asynchronous Cooperation migration strategies, that is, configure multiple processing lines for data migration task set
Journey reads data, by multiple processing threads as multiple queues by individually handling thread as thread is read from source database
Thread is distributed, multiple queue assignment threads distribute the data of reading into a certain data queue, by multiple processing thread conducts
Thread is written, it will be in the data write-in target database in data queue corresponding with write-in thread.
In practical applications, user can select one of which Data Migrating Strategy according to migrating data amount, according to selected
Data Migrating Strategy carry out Data Migration.Wherein, different Data Migrating Strategies is moved suitable for the data of different migration scenes
It moves, for example, it is not especially big that simultaneously and concurrently migration strategy, which is suitable for migrating data amount, migrates the relatively simple Data Migration of logic
Scene；And asynchronous equality migration strategy is suitable for the Data Migration scene that migrating data amount is especially big, data distribution is relatively uniform.
Certainly, those skilled in the art can be according to different Data Migration scenes using the Data Migrating Strategy of adaptation, and the present invention is real
Example is applied not to be restricted this.
For the ease of those skilled in the art understand that the embodiment of the present invention data migration method, extremely below with reference to Fig. 3
The specific example of Fig. 4 illustrates.
Fig. 3 is the data flow schematic diagram of a data migration example of the embodiment of the present invention.It can be seen that source data
There are the tables of data src_log that one includes daily record data in library, wherein including 8 daily record datas, currently by 8 day
Will Data Migration is to target database ds1 to ds4.Wherein, target database ds1 to ds4, each database all have two numbers
According to table, it is named as tar_log_00 successively to tar_log_07.In migrating data, data are submitted to move to data migration server
Set of tasks is moved, data migration task set includes multiple data migration tasks, in one of data migration task, is carried
There are target database mark ds1 to ds4 and Data distribution information [2,2,2,2], hence, it can be determined that target database ds1
It is the current target database for needing that data are written to ds4, moreover, 8 tables of data being written currently is needed to be evenly distributed in 4
In a different target database.Therefore, data migration server can respectively be moved with 8 daily record datas of tables of data src_log
It moves in tables of data tar_log_00 to tar_log_07 8 tables of data.
Fig. 4 is the data flow schematic diagram of another Data Migration example of the embodiment of the present invention.It can be seen that source number
According in library, there are the tables of data src_product that one includes product data, wherein include 5 product information data, when
Before by 5 product information Data Migrations to target data ds1 and ds2.In migrating data, submitted to data migration server
Data migration task set, data migration task set include name of product migration task, product price migration task and product
Inventory migrates task.
In name of product information transfer task, target database is identified as ds1, and Data distribution information is [1], because
This, it may be determined that target database ds1 is the target database for currently needing to be written data, and by tables of data src_product
The id of middle pieces of data and corresponding name of product data are migrated to the tables of data tar_product of target database ds1
In.
In product price information transfer task, target database is identified as ds1, and Data distribution information is [1], because
This migrates the id of pieces of data in tables of data src_product and corresponding product price data to target database
In the tables of data tar_price of ds1.
In product inventory information transfer task, target database is identified as ds1 and ds2, Data distribution information be [1,
3], show to be respectively written into the id of pieces of data in tables of data src_product and corresponding product inventory data to target
In the tables of data tar_stock_1000 of database ds1, and, it is written to three tables of data tar_ of target database ds1
In stock_2000, tar_stock_5000 and tar_stock_now.
According to embodiments of the present invention, by setting Data distribution information, and migrating data is determined according to Data distribution information
Distributing position in multiple target databases and multiple tables of data, by most a database of the Data Migration of source database
Tables of data when, while realizing that data point libraries divides table to migrate, without the migration rules of setting large amount of complex, reduce user
Development and maintenance work, reduce the cost of Data Migration.
Embodiment two
Fig. 5 is the flow chart of data migration method provided by Embodiment 2 of the present invention, and this method takes applied to Data Migration
Business device, the data migration server are configured with multiple processing threads, and this method can specifically include following steps：
Step 210, data migration task set is received；The data migration task set carries source database and identifies, is more
A target database mark and Data distribution information.
Step 220, according to the Data distribution information, target to be written is determined in the multiple target database mark
Database Identification.
Step 230, it is identified in corresponding source database from the source database and reads data to be migrated, and wait moving by described
The data write-in target database to be written is moved to identify in corresponding target database.
Optionally, the method can further include：
When the corresponding migrating data amount of the data migration task set is less than predetermined threshold value, for the Data Migration
The single processing thread of set of tasks configuration；
The step 230, including：
It calls the single processing thread to be identified in corresponding source database from the source database and reads data to be migrated,
And the data write-in target database to be written to be migrated is identified in corresponding target database.
Data migration server can have multiple processing threads, and migrate set of tasks to be handled according to current data
Migrating data amount, carry out processing thread configuration.
In the specific implementation, the corresponding migrating data amount of data migration task set can be compared with predetermined threshold value.
If migrating data amount is less than predetermined threshold value, the single processing thread of data migration task set configuration can be directed to.
When carrying out Data Migration, the single processing thread is called, reads data to be migrated from source database on one side, one
While the data to be migrated of reading are written in each tables of data of target database, read data and write-in data are same by single thread
Step carries out.If having existed simultaneously multiple data migration task set, it is more that this is concurrently handled respectively by multiple processing threads
A Data Migration set of tasks, to achieve the effect that simultaneously and concurrently migrating data.
Above-mentioned data migration method is simultaneously and concurrently migration strategy, is not especially big suitable for migrating data amount, and
Migrate the relatively simple migration task of logic.
Fig. 6 is the schematic diagram of the parallel migration strategy of the embodiment of the present invention.It can be seen that migrating data server receives
To multiple Data Migration operation Job, including migration operation _ 1, migration operation _ 2, migration operation _ 3 and migration operation _ 4.Wherein,
Each Data Migration operation Job may include multiple data migration task TargetTask, that is, a Data Migration operation is one
A Data Migration set of tasks.
Each Data Migration operation Job configurations read data there are one thread, the processing thread is handled from source database,
The data of reading are written to one or more of target database tables of data simultaneously.Multiple Data Migration operation Job, then simultaneously
Row ground executes Data Migration by the processing thread configured.
Optionally, the data migration task set further includes with corresponding data distribution characteristics, the method：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the Data Migration
When the corresponding data distribution characteristics of set of tasks meet default distribution characteristics, configured for the data migration task set single
Thread is handled as reading thread, and, it is queue assignment thread to configure single processing thread, and, configure at least one place
Lineation journey is write-in thread；
The step 230, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, with
And call the queue assignment thread that the data to be migrated of reading are formed multiple data queues, call said write thread will
Data to be migrated in the multiple data queue are sequentially written in the target database to be written and identify corresponding target data
In library.
Data migration task set can have data distribution characteristics, that is, the data to be migrated are in each target database
And the distribution situation in each tables of data, it can be in conjunction with migrating data amount and the data distribution spy of data migration task set
Sign carries out the configuration of processing thread.
In the specific implementation, the corresponding migrating data amount of data migration task set can be compared with predetermined threshold value,
Meanwhile the corresponding data distribution characteristics of data migration task set being compared with default distribution characteristics.If migrating data amount
More than predetermined threshold value, moreover, data distribution characteristics are matched with default distribution characteristics, then the data migration task collection can be directed to
The single processing thread of configuration is closed, as reading thread, and, single processing thread is configured, as queue assignment thread, and,
Multiple processing threads are configured, as write-in thread.
When carrying out Data Migration, single reading thread is called, data to be migrated are read from source database, and,
It calls single queue to distribute thread, distributes the data to be migrated of reading at most a data queue, call multiple write-in threads, it will
Data to be migrated in data queue corresponding with write-in thread, are sequentially written in each tables of data of target database.Queue
Distributing thread equably can distribute data at most a data queue, and data volume to be migrated is impartial in each data queue, and
Each write-in thread is merely responsible for the data write-in in a corresponding data queue, and therefore, each thread that is written is responsible for
The data volume of write-in is impartial.
Above-mentioned data migration method is asynchronous Fair migration strategies, suitable for the larger and each target of migrating data amount
The migration task of the database data volume of being written equalization.
Fig. 7 is the schematic diagram of the asynchronous Fair migration strategies of the embodiment of the present invention.It can be seen that migrating data server
For a Data Migration operation Job configuration, there are one read thread (read threads), a queue assignment thread (split line
Journey) and four write-in threads (write threads), it reads thread and reads data from source database, queue assignment thread will then be read
The data taken are distributed into four data queues (queue), and four write-in threads are then respectively by the data in four data queues
It is written to one or more of target database tables of data.
Optionally, the method further includes：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the Data Migration
When the corresponding data distribution characteristics of set of tasks do not meet default distribution characteristics, configured for the data migration task set single
A processing thread, which is used as, reads thread, and, it is write-in thread to configure multiple processing threads；
The step 230, including：
It calls the reading thread to identify from the source database to read data to be migrated in corresponding source database and incite somebody to action
The data to be migrated read form individual data queue, and, call multiple write-in threads will be in the individual data queue
The data write-in target database to be written to be migrated identifies in corresponding target database.
Relative to above-mentioned asynchronous Fair migration strategies, if migrating data amount is more than predetermined threshold value, still, data distribution is special
When sign is mismatched with default distribution characteristics, it can be directed to the single processing thread of data migration task set configuration, as read line
Journey, and, multiple processing threads are configured, as write-in thread.
When carrying out Data Migration, single reading thread is called, data to be migrated are read from source database, and will read
The data to be migrated taken form individual data queue.Meanwhile multiple write-in threads are called, it will be to be migrated in individual data queue
Data are written in each tables of data of target database.
In above-mentioned data migration process, queue assignment thread is not configured, reads thread as the producer, write line
Cheng Zuowei consumer, the data resource that multiple write-in threads are produced to reading thread, is consumed, respectively in a manner of seizing consumption
A write-in thread can write data into the tables of data of all databases, even if distribution of the data in each tables of data
It is uneven, will not influence it is each write-in thread write-in processing, avoid due to data unevenness with or some write-in thread property
The problem of energy problem causes overall data write efficiency to decline.
Above-mentioned data migration method is asynchronous Crazy migration strategies, larger suitable for migrating data amount, and each target
The unequal migration task of the database data volume of being written.
Fig. 8 is the schematic diagram of the asynchronous Crazy migration strategies of the embodiment of the present invention.It can be seen that migrating data service
There are one read thread (read threads) and three write-in thread (write lines for a Data Migration operation Job configuration for device
Journey), it reads thread and reads data from source database, and the data of reading are formed into a data queue (queue), three are write
Enter thread then data in the data queue are written to one or more of target database data in a manner of seizing consumption
Table.
Optionally, the data migration task set further includes with corresponding data manipulation amount, the method：
When the corresponding data manipulation amount of the data migration task set is more than predetermined threshold value, for the Data Migration
The single processing thread of set of tasks configuration, which is used as, reads thread, and, it is queue assignment thread to configure multiple processing threads, with
And at least one processing thread of configuration is write-in thread；
The step 230, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, with
And call multiple queue assignment threads that the data to be migrated of reading are formed multiple data queues, call said write thread will
Data to be migrated in the multiple data queue are sequentially written in the target database to be written and identify corresponding target data
In library.
In addition to migrating data amount and data distribution characteristics, data migration task set can also have corresponding data manipulation
To the quantity of the data operation of being executed when amount, i.e. execution Data Migration.It can be grasped according to the data of data migration task set
It measures, carries out the configuration of processing thread.
In the specific implementation, the corresponding data manipulation amount of data migration task set can be compared with predetermined threshold value.
If data manipulation amount is more than predetermined threshold value, it can be directed to the single processing thread of data migration task set configuration, as reading
Thread, and, multiple processing threads are configured, as queue assignment thread, and, multiple processing threads are configured, as write line
Journey.
When carrying out Data Migration, single reading thread is called, data to be migrated are read from source database, and,
Multiple queue assignment threads are called, respectively by the data to be migrated of reading, distribute at most a data queue, and, it calls multiple
Thread is written, by the data to be migrated in data queue corresponding with write-in thread, is sequentially written in each number of target database
According in table.
When distributing data to data queue, the data manipulations such as complicated subquery, calculating, fractionation are may relate to.
In above-mentioned data migration process, the data for being responsible for handling data queue by multiple queue assignment threads are distributed, even if in the presence of
The data manipulations such as complicated subquery, calculating, fractionation, still can ensure that data are distributed to the processing speed of data queue,
The data manipulation speed that avoids single queue distribution thread process complexity is excessively slow, leads to asking for entire data migration efficiency reduction
Topic.
Above-mentioned data migration method is asynchronous Cooperation migration strategies, relatively small suitable for main table data volume,
But need to carry out the data migration task of the data manipulations such as complicated subquery, calculating, fractionation.
Fig. 9 is the schematic diagram of the asynchronous Crazy migration strategies of the embodiment of the present invention.It can be seen that migrating data service
Device is directed to a Data Migration operation Job, and there are one read thread (read threads), four queue assignment thread (split for configuration
Thread) and two write-in threads (write threads).It reads thread and reads data from source database, and by the data shape of reading
At an initial data queue, four queue assignment threads for the data in initial data queue, carry out subquery, calculating,
The data manipulations such as fractionation, and distribute to two object queue threads, two write-in threads, then respectively by corresponding target data
Data in queue, write-in to one or more of target database tables of data.
In practical applications, user can select according to factors such as migrating data amount, data distribution characteristics, data manipulation amounts
Determine one of which Data Migrating Strategy, Data Migration is carried out according to selected Data Migrating Strategy.To which user can be directed to number
According to migration task, Data Migration is carried out using the Data Migrating Strategy of adaptation, improves the efficiency of Data Migration.
Optionally, the data to be migrated include multiple subdatas, described that the data write-in to be migrated is described to be written
Enter target database and identify step in corresponding target database, can specifically include：
The multiple subdata batch write-in to the target database to be written is identified into corresponding target database.
In actual Data Migration scene, data migration server can generate the connection of target database, and for company
Deliver a child into session.The processing thread of data migration server reads data from source database line by line, and (each row of data is a son
Data), and the data of reading are put into session line by line, after reaching preset submission data amount values, then carry out
The batch of commit is submitted, and batch data is written into each tables of data of target database.
Step 240, when receiving the write-in notice extremely that the target database is sent, from the multiple subdata
Extract at least one subdata.
Step 250, at least one subdata is written to the target database to be written and identifies corresponding target
Database, and judge whether at least one subdata is successfully written.
Step 260, if so, the step of extracting at least one subdata from the multiple subdata described in returning.
Step 270, it is unsuccessfully identified if it is not, being then directed at least one subdata addition write-in.
In actual Data Migration scene, it may occur that the write-in of target database is abnormal.More specifically, batch is write
When entering data to target database, database level can report an error, such as non-empty field inserts null value, field length overlength, master
Key conflict etc. reports an error, and target database can correspondingly send the abnormal notice of write-in to data migration server.Therefore, it is receiving
When the write-in that target database is sent out notifies extremely, from multiple subdatas of data to be migrated, single subdata is extracted, by this
Single subdata is written to target database, judges whether the single subdata write-in database succeeds.If be written successfully,
Next single subdata is continued to write to, if write-in is unsuccessful, is unsuccessfully identified for subdata addition write-in.Practical application
In, the subdata added with write-in failure mark can be in addition stored, so that subdata re-writes by the part in subsequent set
Database.
In practical applications, when database level inserts null value, field length overlength, major key conflict for non-empty field
Situations such as report an error, data migration server can be handled with this batch submission of rollback, will originally need batch submit number
According to, single submission by way of submitted, abnormal data is searched during single is submitted, and abnormal data is preserved
To specific database, then, when submitting data next time, reverts to batch and submit.
According to embodiments of the present invention, by that when receiving the write-in notice extremely that the target database is sent, will criticize
Amount submits data to be converted to single submission data, determines abnormal data when individually submitting data, is kept away using data rewind mechanism
The missing data in Data Migration is exempted from, ensure that all data can completely be migrated to target database, improve number
According to the reliability of migration.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the method further includes：
When receiving data breakpoint migration task, the maximum data ident value in the multiple tables of data is searched, as
Target data break value；
Breakpoint transmission migration process is executed according to the target data break value.
It should be noted that in actual Data Migration scene, it, can in the implementation procedure of some data migration task
Data Migration can be caused to interrupt there is a situation where server delay machine, therefore, it is necessary to carry out breakpoint migration.It is moved carrying out breakpoint
When shifting, it is ensured that on the basis of upper primary data be written, continue to write to follow-up data, cannot be repeatedly written, can not lose
Fail to write into.
It can be with it should be further noted that each target database may include multiple tables of data, in each tables of data
Including multiple data, by already existing data in tables of data, write-in data are named as in order to distinguish explanation.Wherein, number is written
According to Data Identification value, the Data Identification value is for sequence of positions of the mark data in tables of data.For example, write-in data
Data Identification value can be the data ID such as 20130621,20130622.
In the specific implementation, when receiving data breakpoint migration task, it can be directed to the Data Identification value of multiple tables of data,
The maximum Data Identification value of numerical value is searched, by the Data Identification value, as target data break value.Then, according to the number of targets
According to break value, breakpoint transmission migration process is executed.
For example, it is 20130628 to find the maximum Data Identification value of numerical value, and it therefore, can be in data to be migrated, mistake
Filter out Data Identification value be more than 20130628 multiple targets data to be migrated, and by multiple target data to be migrated be written to
In multiple tables of data of target database.
It should be noted that above-mentioned breakpoint moving method is suitable for the data migration method based on synchronous migration strategy.
In synchronous migration strategy, only there are one processing threads to be responsible for data write-in tables of data, when which is sequentially written data,
Write sequence is consistent with the Data Identification value of data, therefore, can judge data break value according to Data Identification value and execute disconnected
Point resumes migration process.
Figure 10 is a kind of schematic diagram of breakpoint moving method of the embodiment of the present invention.It can be seen that certain of source database
Include multiple data in a tables of data, the data ID of which part data is respectively 20130621,20130622 ... N.
It is abnormal when the data of source database are written to multiple tables of data of target database and needs to execute data breakpoint migration times
Business searches the maximum number of numerical value at this point, for multiple tables of data such as object table _ 1, object table _ 2, object table _ 3 and object table _ 4
According to ID, 20130628 in object table _ 2 have been obtained as maximum data ID, have determined that it is target data break value.According to number of targets
According to break value 20130628, in source database, determine that the data ID more than target data break value 20130628 is
20130629,20130630 ... N filters out corresponding data, and executes breakpoint transmission migration process.
According to embodiments of the present invention, disconnected as target data by searching for the maximum data ident value in multiple tables of data
Point value, and breakpoint transmission migration process is executed according to target data break value, in the Data Migration mistake using synchronous migration strategy
Cheng Zhong may be implemented the breakpoint transmission of data, avoid loss of data.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the method further includes：
When receiving data breakpoint migration task, its maximum data mark is searched respectively in the multiple tables of data
Value, as multiple candidate data break values；
In the multiple candidate data break value, minimum candidate data break value is searched, as target data break value；
Breakpoint transmission migration process is executed according to the target data break value.
In the specific implementation, when receiving data breakpoint migration task, it can be directed to each tables of data, it is maximum to search numerical value
Data Identification value, as candidate data break value.Then, in multiple candidate data break values, the time of numerical value minimum is searched
Data breakpoint value is selected, breakpoint transmission migration process is executed according to the target data break value as target data break value.
It should be noted that above-mentioned breakpoint moving method is suitable for the data migration method based on asynchronous migratory strategy.
In asynchronous migratory strategy, there are multiple processing threads to be responsible for data write-in tables of data, when data are sequentially written in processing thread, each
The migration progress of tables of data is different, and the Data Identification value of write sequence and data is not consistent, can not sentence according to Data Identification value
Fixed number is according to break value.Therefore, it is necessary to first determine the maximum data ident value in each tables of data as candidate data break value, so
Afterwards again in multiple candidate data break values, the conduct target data break value of numerical value minimum is searched.
Figure 11 is the schematic diagram of another breakpoint moving method of the embodiment of the present invention.It can be seen that source database
Include multiple data in some tables of data, the data ID of which part data is respectively 20130621,20130622 ... N.
It is abnormal when being written the data of source database to multiple tables of data of target database and needs to execute data breakpoint migration
Task, wherein need the data for being 20130621,20130625,20130629 by data ID that object table _ 1 is written, and be written
It is abnormal after the data that data ID is 20130621, it is that 20130625 and 20130629 data are not write to lead to data ID
Enter object table _ 1.At this point, for multiple tables of data such as object table _ 1, object table _ 2, object table _ 3 and object table _ 4, numerical value is searched
Maximum data ID, respectively 20130621,20130626,20130627 and 20130628.If it is determined that numerical value is maximum
20130628 be target data break value, and the data that data ID is 20130625 will not then be performed breakpoint transmission migration, cause
Loss of data.Therefore, it is necessary in data ID 20130621,20130626,20130627 and 20130628, search numerical value most
Small data ID obtains 20130621 as minimum data ID, therefore is used as target data break value by 20130621.Based on mesh
It marks data breakpoint value 20130621 and executes breakpoint transmission migration process, then the data ID data for being 20130625 can be also written
To database, avoid that loss of data occurs.
According to embodiments of the present invention, disconnected as candidate data by searching for the maximum data ident value in multiple tables of data
Point value, and minimum candidate data break value is searched in multiple candidate data break values, as target data break value, root
Breakpoint transmission migration process is executed according to the target data break value, it, can in the data migration process using asynchronous migratory strategy
To realize the breakpoint transmission of data, loss of data is avoided.
Embodiment three
Figure 12 is the structure diagram for the data migration server that the embodiment of the present invention three provides, the data migration server
300, it can specifically include with lower module：
Task receiving module 310, for receiving data migration task set；The data migration task set carries source number
According to library mark, multiple target databases mark and Data distribution information；
Determining module 320 is identified, is used for according to the Data distribution information, in the multiple target database mark really
Fixed target database mark to be written；
Data Migration module 330 reads data to be migrated for being identified in corresponding source database from the source database,
And the data write-in target database to be written to be migrated is identified in corresponding target database.
According to embodiments of the present invention, by setting Data distribution information, and migrating data is determined according to Data distribution information
Distributing position in multiple target databases and multiple tables of data, by most a database of the Data Migration of source database
Tables of data when, while realizing that data point libraries divides table to migrate, without the migration rules of setting large amount of complex, reduce user
Development and maintenance work, reduce the cost of Data Migration.
Example IV
Figure 13 is the structure diagram for the data migration server that the embodiment of the present invention four provides, the data migration server
400, it can specifically include with lower module：
Task receiving module 410, for receiving data migration task set；The data migration task set carries source number
According to library mark, multiple target databases mark and Data distribution information；
Determining module 420 is identified, is used for according to the Data distribution information, in the multiple target database mark really
Fixed target database mark to be written；
Data Migration module 430 reads data to be migrated for being identified in corresponding source database from the source database,
And the data write-in target database to be written to be migrated is identified in corresponding target database.
Optionally, the data to be migrated include multiple subdatas, and the Data Migration module 430 can specifically include：
Submodule is written in batch data, for being written the multiple subdata batch to the target database to be written
Identify corresponding target database.
Subdata extraction module 440, for when receiving the write-in that the target database is sent and notifying extremely, from institute
It states and extracts at least one subdata in multiple subdatas；
Judgment module 450 is written in data, for being written at least one subdata to the target data to be written
Library identifies corresponding target database, and judges whether at least one subdata is successfully written；If so, execute, if it is not,
Then execute；
Module 460 is returned to, for returning to the subdata extraction module 440；
Write-in unsuccessfully identifies add module 470, is unsuccessfully identified for being directed at least one subdata addition write-in.
Optionally, the server further includes：
First processing thread configuration module, for being less than in advance when the corresponding migrating data amount of the data migration task set
If when threshold value, single processing thread is configured for the data migration task set；
The Data Migration module 430, including：
Submodule is simultaneously and concurrently migrated, for calling the single processing thread to identify corresponding source from the source database
Data to be migrated are read in database, and the data write-in target database to be written to be migrated is identified into corresponding mesh
It marks in database.
Optionally, the data migration task set further includes with corresponding data distribution characteristics, the server：
Second processing thread configuration module, for being more than institute when the corresponding migrating data amount of the data migration task set
Predetermined threshold value is stated and when the corresponding data distribution characteristics of the data migration task set meet default distribution characteristics, for institute
The single processing thread of data migration task set configuration is stated as reading thread, and, single processing thread is configured as queue point
With thread, and, it is write-in thread to configure at least one processing thread；
The Data Migration module 430, including：
Asynchronous impartial migration submodule, for calling the reading thread to identify corresponding source data from the source database
Data to be migrated are read in library, and, call the queue assignment thread that the data to be migrated of reading are formed multiple data teams
Row call said write thread that the data to be migrated in the multiple data queue are sequentially written in the target data to be written
Library identifies in corresponding target database.
Optionally, the server further includes：
Third handles thread configuration module, for being more than institute when the corresponding migrating data amount of the data migration task set
State predetermined threshold value and when the corresponding data distribution characteristics of the data migration task set do not meet default distribution characteristics, for
The single processing thread of data migration task set configuration, which is used as, reads thread, and, it is write-in to configure multiple processing threads
Thread；
The Data Migration module 430, including：
It is asynchronous to seize migration submodule, for calling the reading thread to identify corresponding source data from the source database
Data to be migrated are read in library and the data to be migrated of reading are formed into individual data queue, and, call multiple write-in threads
The data to be migrated write-in target database to be written in the individual data queue is identified into corresponding target database
In.
Optionally, the data migration task set further includes with corresponding data manipulation amount, the server：
Fourth process thread configuration module, for being more than in advance when the corresponding data manipulation amount of the data migration task set
If when threshold value, single processing thread is configured as reading thread for the data migration task set, and, configure multiple places
Lineation journey is queue assignment thread, and, it is write-in thread to configure at least one processing thread；
The Data Migration module 430, including：
Asynchronous cooperation migrates submodule, for calling the reading thread to identify corresponding source data from the source database
Data to be migrated are read in library, and, call multiple queue assignment threads that the data to be migrated of reading are formed multiple data teams
Row call said write thread that the data to be migrated in the multiple data queue are sequentially written in the target data to be written
Library identifies in corresponding target database.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the server further includes：
First object data breakpoint value determining module, for when receiving data breakpoint migration task, searching described more
Maximum data ident value in a tables of data, as target data break value；
First breakpoint transmission migration process execution module is moved for executing breakpoint transmission according to the target data break value
Shifting is handled.
Optionally, the target database includes multiple tables of data, and the tables of data includes multiple write-in data, described
Data are written with Data Identification value, the server further includes：
Candidate point value determining module, for when receive data breakpoint migrate task when, in the multiple tables of data
Its maximum data ident value is searched respectively, as multiple candidate data break values；
Second target data break value determining module, in the multiple candidate data break value, searching minimum candidate
Data breakpoint value, as target data break value；
Second breakpoint transmission migration process execution module is moved for executing breakpoint transmission according to the target data break value
Shifting is handled.
According to embodiments of the present invention, by that when receiving the write-in notice extremely that the target database is sent, will criticize
Amount submits data to be converted to single submission data, determines abnormal data when individually submitting data, is kept away using data rewind mechanism
The missing data in Data Migration is exempted from, ensure that all data can completely be migrated to target database, improve number
According to the reliability of migration.
According to embodiments of the present invention, disconnected as target data by searching for the maximum data ident value in multiple tables of data
Point value, and breakpoint transmission migration process is executed according to target data break value, in the Data Migration mistake using synchronous migration strategy
Cheng Zhong may be implemented the breakpoint transmission of data, avoid loss of data.
According to embodiments of the present invention, disconnected as candidate data by searching for the maximum data ident value in multiple tables of data
Point value, and minimum candidate data break value is searched in multiple candidate data break values, as target data break value, root
Breakpoint transmission migration process is executed according to the target data break value, it, can in the data migration process using asynchronous migratory strategy
To realize the breakpoint transmission of data, loss of data is avoided.
For server example, since it is basically similar to the method embodiment, so fairly simple, the phase of description
Place is closed referring to the part of embodiment of the method to illustrate.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with
The difference of other embodiment, the same or similar parts between the embodiments can be referred to each other.
It should be understood by those skilled in the art that, the embodiment of the embodiment of the present invention can be provided as method, apparatus or calculate
Machine program product.Therefore, the embodiment of the present invention can be used complete hardware embodiment, complete software embodiment or combine software and
The form of the embodiment of hardware aspect.Moreover, the embodiment of the present invention can be used one or more wherein include computer can
With in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of program code
The form of the computer program product of implementation.
In a typical configuration, the computer equipment includes one or more processors (CPU), input/output
Interface, network interface and memory.Memory may include the volatile memory in computer-readable medium, random access memory
The forms such as device (RAM) and/or Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is to calculate
The example of machine readable medium.Computer-readable medium includes that permanent and non-permanent, removable and non-removable media can be with
Information storage is realized by any method or technique.Information can be computer-readable instruction, data structure, the module of program or
Other data.The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM
(SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only memory
(ROM), electrically erasable programmable read-only memory (EEPROM), fast flash memory bank or other memory techniques, CD-ROM are read-only
Memory (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic tape cassette, tape magnetic disk storage or
Other magnetic storage apparatus or any other non-transmission medium can be used for storage and can be accessed by a computing device information.According to
Herein defines, and computer-readable medium does not include non-persistent computer readable media (transitory media), such as
The data-signal and carrier wave of modulation.
The embodiment of the present invention be with reference to according to the method for the embodiment of the present invention, terminal device (system) and computer program
The flowchart and/or the block diagram of product describes.It should be understood that flowchart and/or the block diagram can be realized by computer program instructions
In each flow and/or block and flowchart and/or the block diagram in flow and/or box combination.These can be provided
Computer program instructions are set to all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing terminals
Standby processor is to generate a machine so that is held by the processor of computer or other programmable data processing terminal equipments
Capable instruction generates for realizing in one flow of flow chart or multiple flows and/or one box of block diagram or multiple boxes
The device of specified function.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing terminal equipments
In computer-readable memory operate in a specific manner so that instruction stored in the computer readable memory generates packet
The manufacture of command device is included, which realizes in one flow of flow chart or multiple flows and/or one side of block diagram
The function of being specified in frame or multiple boxes.
These computer program instructions can be also loaded into computer or other programmable data processing terminal equipments so that
Series of operation steps are executed on computer or other programmable terminal equipments to generate computer implemented processing, thus
The instruction executed on computer or other programmable terminal equipments is provided for realizing in one flow of flow chart or multiple flows
And/or in one box of block diagram or multiple boxes specify function the step of.
Although the preferred embodiment of the embodiment of the present invention has been described, once a person skilled in the art knows bases
This creative concept, then additional changes and modifications can be made to these embodiments.So the following claims are intended to be interpreted as
Including preferred embodiment and fall into all change and modification of range of embodiment of the invention.
Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that process, method, article or terminal device including a series of elements not only wrap
Those elements are included, but also include other elements that are not explicitly listed, or further include for this process, method, article
Or the element that terminal device is intrinsic.In the absence of more restrictions, being wanted by what sentence "including a ..." limited
Element, it is not excluded that there is also other identical elements in process, method, article or the terminal device including the element.
Claims (10)
1. a kind of data migration method is applied to data migration server, which is characterized in that the method includes：
Receive data migration task set；The data migration task set carries source database mark, multiple target databases
Mark and Data distribution information；
According to the Data distribution information, target database mark to be written is determined in the multiple target database mark；
It is identified from the source database and reads data to be migrated in corresponding source database, and institute is written into the data to be migrated
Target database to be written is stated to identify in corresponding target database.
2. according to the method described in claim 1, it is characterized in that, the method further includes：
When the corresponding migrating data amount of the data migration task set is less than predetermined threshold value, for the data migration task
The single processing thread of set configuration；
Described identified in corresponding source database from the source database reads data to be migrated, and the data to be migrated are write
Enter the target database to be written and identifies step in corresponding target database, including：
It calls the single processing thread to be identified from the source database and reads data to be migrated in corresponding source database, and will
The data write-in target database to be written to be migrated identifies in corresponding target database.
3. according to the method described in claim 1, it is characterized in that, the data migration task set is with corresponding data point
Cloth feature, the method further include：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the data migration task
When gathering corresponding data distribution characteristics and meeting default distribution characteristics, single processing is configured for the data migration task set
Thread, which is used as, reads thread, and, it is queue assignment thread to configure single processing thread, and, configure at least one processing line
Journey is write-in thread；
Described identified in corresponding source database from the source database reads data to be migrated, and the data to be migrated are write
Enter the target database to be written and identifies step in corresponding target database, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, and, it adjusts
The data to be migrated of reading are formed into multiple data queues with the queue assignment thread, call said write thread will be described more
Data to be migrated in a data queue are sequentially written in the target database to be written and identify in corresponding target database.
4. according to the method described in claim 3, it is characterized in that, the method further includes：
When the corresponding migrating data amount of the data migration task set is more than the predetermined threshold value and the data migration task
When gathering corresponding data distribution characteristics and not meeting default distribution characteristics, single place is configured for the data migration task set
Lineation journey, which is used as, reads thread, and, it is write-in thread to configure multiple processing threads；
Described identified in corresponding source database from the source database reads data to be migrated, and the data to be migrated are write
Enter the target database to be written and identifies step in corresponding target database, including：
It calls the reading thread to identify from the source database to read data to be migrated in corresponding source database and will read
Data to be migrated form individual data queue, and, call multiple write-in threads that will wait moving in the individual data queue
The data write-in target database to be written is moved to identify in corresponding target database.
5. according to the method described in claim 1, it is characterized in that, the data migration task set is grasped with corresponding data
It measures, the method further includes：
When the corresponding data manipulation amount of the data migration task set is more than predetermined threshold value, for the data migration task
The single processing thread of set configuration, which is used as, reads thread, and, it is queue assignment thread to configure multiple processing threads, and, match
It is write-in thread to set at least one processing thread；
Described identified in corresponding source database from the source database reads data to be migrated, and the data to be migrated are write
Enter the target database to be written and identifies step in corresponding target database, including：
It calls the reading thread to be identified in corresponding source database from the source database and reads data to be migrated, and, it adjusts
The data to be migrated of reading are formed into multiple data queues with multiple queue assignment threads, call said write thread will be described more
Data to be migrated in a data queue are sequentially written in the target database to be written and identify in corresponding target database.
6. according to the method described in claim 1, it is characterized in that, the data to be migrated include multiple subdatas, it is described will
The data to be migrated are written the target database to be written and identify step in corresponding target database, including：
The multiple subdata batch write-in to the target database to be written is identified into corresponding target database.
7. according to the method described in claim 6, it is characterized in that, the multiple subdata batch is written to described described
After target database to be written identifies the step of corresponding target database, the method further includes：
When receiving the write-in notice extremely that the target database is sent, extracted from the multiple subdata at least one
Subdata；
At least one subdata is written to the target database to be written and identifies corresponding target database, and is judged
Whether at least one subdata is successfully written；
If so, the step of extracting at least one subdata from the multiple subdata described in returning；
It is unsuccessfully identified if it is not, being then directed at least one subdata addition write-in.
8. according to the method described in claim 2, it is characterized in that, the target database includes multiple tables of data, the number
Include multiple write-in data according to table, said write data further include with Data Identification value, the method：
When receiving data breakpoint migration task, the maximum data ident value in the multiple tables of data is searched, as target
Data breakpoint value；
Breakpoint transmission migration process is executed according to the target data break value.
9. the method according to claim 3 or 5, which is characterized in that the target database includes multiple tables of data, described
Tables of data includes multiple write-in data, and said write data further include with Data Identification value, the method：
When receiving data breakpoint migration task, searches its maximum data ident value respectively in the multiple tables of data, make
For multiple candidate data break values；
In the multiple candidate data break value, minimum candidate data break value is searched, as target data break value；
Breakpoint transmission migration process is executed according to the target data break value.
10. a kind of data migration server, which is characterized in that the server includes：
Task receiving module, for receiving data migration task set；The data migration task set carries source database mark
Know, multiple target databases identify and Data distribution information；
Determining module is identified, for according to the Data distribution information, determination to be to be written in the multiple target database mark
Enter target database mark；
Data Migration module reads data to be migrated for being identified from the source database in corresponding source database, and by institute
The data write-in target database to be written to be migrated is stated to identify in corresponding target database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810002213.6A CN108280148A (en) | 2018-01-02 | 2018-01-02 | A kind of data migration method and data migration server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810002213.6A CN108280148A (en) | 2018-01-02 | 2018-01-02 | A kind of data migration method and data migration server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108280148A true CN108280148A (en) | 2018-07-13 |
Family
ID=62802989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810002213.6A Pending CN108280148A (en) | 2018-01-02 | 2018-01-02 | A kind of data migration method and data migration server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108280148A (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241027A (en) * | 2018-07-23 | 2019-01-18 | 平安科技（深圳）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of Data Migration |
| CN109241029A (en) * | 2018-08-08 | 2019-01-18 | 新智能源系统控制有限责任公司 | A kind of method and apparatus for realizing SCADA system data library smooth migration |
| CN109460439A (en) * | 2018-11-14 | 2019-03-12 | 泰康保险集团股份有限公司 | A kind of data processing method, device, medium and electronic equipment |
| CN109710624A (en) * | 2018-12-19 | 2019-05-03 | 泰康保险集团股份有限公司 | Data processing method, device, medium and electronic equipment |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| CN110209650A (en) * | 2019-05-05 | 2019-09-06 | 苏宁易购集团股份有限公司 | The regular moving method of data, device, computer equipment and storage medium |
| CN110555014A (en) * | 2019-09-06 | 2019-12-10 | 中国联合网络通信集团有限公司 | Data migration method and system, electronic device and storage medium |
| CN111258990A (en) * | 2020-02-17 | 2020-06-09 | 同盾控股有限公司 | Index database data migration method, device, equipment and storage medium |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111723079A (en) * | 2020-06-28 | 2020-09-29 | 深圳壹账通智能科技有限公司 | Data migration method and device, computer equipment and storage medium |
| CN111831625A (en) * | 2020-07-14 | 2020-10-27 | 深圳力维智联技术有限公司 | Data migration method, data migration device and readable storage medium |
| CN111930715A (en) * | 2020-07-16 | 2020-11-13 | 北京金山云网络技术有限公司 | Data migration method and device, computer equipment and storage medium |
| CN112000641A (en) * | 2020-07-31 | 2020-11-27 | 新华三大数据技术有限公司 | Data storage method, device and system |
| CN112000292A (en) * | 2020-08-21 | 2020-11-27 | 北京金山云网络技术有限公司 | Data migration method and device, electronic equipment and computer storage medium |
| CN112632032A (en) * | 2020-12-18 | 2021-04-09 | 华人运通(上海)云计算科技有限公司 | Data migration method and device, storage medium and terminal equipment |
| CN113051245A (en) * | 2019-12-26 | 2021-06-29 | 云丁网络技术(北京)有限公司 | Method, device and system for migrating data |
| CN113268466A (en) * | 2021-06-07 | 2021-08-17 | 上海数禾信息科技有限公司 | Method and system for smoothly migrating message cluster |
| CN113360495A (en) * | 2021-04-20 | 2021-09-07 | 山东英信计算机技术有限公司 | Database query interruption recovery method, device, equipment and readable medium |
| CN113704277A (en) * | 2021-07-14 | 2021-11-26 | 浪潮商用机器有限公司 | Database-based breakpoint continuous transmission method and related device |
| CN114077602A (en) * | 2022-01-13 | 2022-02-22 | 中兴通讯股份有限公司 | Data migration method and device, electronic equipment and storage medium |
| WO2022062594A1 (en) * | 2020-09-22 | 2022-03-31 | 华为云计算技术有限公司 | Method and apparatus for migrating database, computing device and computer program product |
| CN115941677A (en) * | 2022-09-28 | 2023-04-07 | 电子科技大学 | Breakpoint continuous transmission method for sub-table migration of database |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793424A (en) * | 2012-10-31 | 2014-05-14 | 阿里巴巴集团控股有限公司 | Database data migration method and database data migration system |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| CN105989006A (en) * | 2015-01-27 | 2016-10-05 | 中芯国际集成电路制造(上海)有限公司 | Data migration method and device |
| CN106294440A (en) * | 2015-05-27 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN107153662A (en) * | 2016-03-04 | 2017-09-12 | 华为技术有限公司 | A kind of data processing method and device |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107391755A (en) * | 2017-08-21 | 2017-11-24 | 中国银联股份有限公司 | A kind of data distribution regulation, the method and apparatus of inquiry |
-
2018
- 2018-01-02 CN CN201810002213.6A patent/CN108280148A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103793424A (en) * | 2012-10-31 | 2014-05-14 | 阿里巴巴集团控股有限公司 | Database data migration method and database data migration system |
| CN105989006A (en) * | 2015-01-27 | 2016-10-05 | 中芯国际集成电路制造(上海)有限公司 | Data migration method and device |
| CN106294440A (en) * | 2015-05-27 | 2017-01-04 | 阿里巴巴集团控股有限公司 | The method and apparatus of data real-time migration |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| CN107153662A (en) * | 2016-03-04 | 2017-09-12 | 华为技术有限公司 | A kind of data processing method and device |
| CN107357883A (en) * | 2017-06-30 | 2017-11-17 | 北京奇虎科技有限公司 | Data migration method and device |
| CN107391755A (en) * | 2017-08-21 | 2017-11-24 | 中国银联股份有限公司 | A kind of data distribution regulation, the method and apparatus of inquiry |
Cited By (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241027B (en) * | 2018-07-23 | 2023-11-07 | 平安科技（深圳）有限公司 | Data migration method, device, electronic equipment and computer readable storage medium |
| CN109241027A (en) * | 2018-07-23 | 2019-01-18 | 平安科技（深圳）有限公司 | Method, apparatus, electronic equipment and the computer readable storage medium of Data Migration |
| CN109241029B (en) * | 2018-08-08 | 2020-09-25 | 新智能源系统控制有限责任公司 | Method and device for realizing smooth migration of SCADA system database |
| CN109241029A (en) * | 2018-08-08 | 2019-01-18 | 新智能源系统控制有限责任公司 | A kind of method and apparatus for realizing SCADA system data library smooth migration |
| CN109460439A (en) * | 2018-11-14 | 2019-03-12 | 泰康保险集团股份有限公司 | A kind of data processing method, device, medium and electronic equipment |
| CN109710624A (en) * | 2018-12-19 | 2019-05-03 | 泰康保险集团股份有限公司 | Data processing method, device, medium and electronic equipment |
| CN109710624B (en) * | 2018-12-19 | 2021-06-11 | 泰康保险集团股份有限公司 | Data processing method, device, medium and electronic equipment |
| CN110209650B (en) * | 2019-05-05 | 2021-06-29 | 苏宁易购集团股份有限公司 | Data normalization and migration method and device, computer equipment and storage medium |
| CN110209650A (en) * | 2019-05-05 | 2019-09-06 | 苏宁易购集团股份有限公司 | The regular moving method of data, device, computer equipment and storage medium |
| CN110196884B (en) * | 2019-05-31 | 2022-04-29 | 北京大米科技有限公司 | Data writing method based on distributed database, storage medium and electronic equipment |
| CN110196884A (en) * | 2019-05-31 | 2019-09-03 | 北京大米科技有限公司 | Method for writing data, storage medium and electronic equipment based on distributed data base |
| CN110555014A (en) * | 2019-09-06 | 2019-12-10 | 中国联合网络通信集团有限公司 | Data migration method and system, electronic device and storage medium |
| CN110555014B (en) * | 2019-09-06 | 2022-04-15 | 中国联合网络通信集团有限公司 | Data migration method and system, electronic device and storage medium |
| CN113051245A (en) * | 2019-12-26 | 2021-06-29 | 云丁网络技术(北京)有限公司 | Method, device and system for migrating data |
| CN111258990B (en) * | 2020-02-17 | 2023-04-07 | 同盾控股有限公司 | Index database data migration method, device, equipment and storage medium |
| CN111258990A (en) * | 2020-02-17 | 2020-06-09 | 同盾控股有限公司 | Index database data migration method, device, equipment and storage medium |
| CN111324596A (en) * | 2020-03-06 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111324596B (en) * | 2020-03-06 | 2021-06-11 | 腾讯科技（深圳）有限公司 | Data migration method and device for database cluster and electronic equipment |
| CN111723079A (en) * | 2020-06-28 | 2020-09-29 | 深圳壹账通智能科技有限公司 | Data migration method and device, computer equipment and storage medium |
| CN111831625A (en) * | 2020-07-14 | 2020-10-27 | 深圳力维智联技术有限公司 | Data migration method, data migration device and readable storage medium |
| CN111831625B (en) * | 2020-07-14 | 2024-03-12 | 深圳力维智联技术有限公司 | Data migration method, data migration device, and readable storage medium |
| CN111930715A (en) * | 2020-07-16 | 2020-11-13 | 北京金山云网络技术有限公司 | Data migration method and device, computer equipment and storage medium |
| CN112000641B (en) * | 2020-07-31 | 2022-08-30 | 新华三大数据技术有限公司 | Data storage method, device and system |
| CN112000641A (en) * | 2020-07-31 | 2020-11-27 | 新华三大数据技术有限公司 | Data storage method, device and system |
| CN112000292A (en) * | 2020-08-21 | 2020-11-27 | 北京金山云网络技术有限公司 | Data migration method and device, electronic equipment and computer storage medium |
| WO2022062594A1 (en) * | 2020-09-22 | 2022-03-31 | 华为云计算技术有限公司 | Method and apparatus for migrating database, computing device and computer program product |
| CN112632032B (en) * | 2020-12-18 | 2022-12-27 | 华人运通(上海)云计算科技有限公司 | Data migration method and device, storage medium and terminal equipment |
| CN112632032A (en) * | 2020-12-18 | 2021-04-09 | 华人运通(上海)云计算科技有限公司 | Data migration method and device, storage medium and terminal equipment |
| CN113360495B (en) * | 2021-04-20 | 2023-02-24 | 山东英信计算机技术有限公司 | Database query interruption recovery method, device, equipment and readable medium |
| CN113360495A (en) * | 2021-04-20 | 2021-09-07 | 山东英信计算机技术有限公司 | Database query interruption recovery method, device, equipment and readable medium |
| CN113268466A (en) * | 2021-06-07 | 2021-08-17 | 上海数禾信息科技有限公司 | Method and system for smoothly migrating message cluster |
| CN113704277A (en) * | 2021-07-14 | 2021-11-26 | 浪潮商用机器有限公司 | Database-based breakpoint continuous transmission method and related device |
| CN113704277B (en) * | 2021-07-14 | 2024-02-02 | 浪潮商用机器有限公司 | Breakpoint continuous transmission method based on database and related device |
| CN114077602B (en) * | 2022-01-13 | 2022-05-17 | 中兴通讯股份有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN114077602A (en) * | 2022-01-13 | 2022-02-22 | 中兴通讯股份有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN115941677A (en) * | 2022-09-28 | 2023-04-07 | 电子科技大学 | Breakpoint continuous transmission method for sub-table migration of database |
| CN115941677B (en) * | 2022-09-28 | 2024-03-19 | 电子科技大学 | Breakpoint continuous transmission method for database sub-table migration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280148A (en) | A kind of data migration method and data migration server | |
| AU2020200578B2 (en) | Intelligent configuration discovery techniques | |
| US11943104B2 (en) | Application migration system | |
| US20210279241A1 (en) | Splitting a time-range query into multiple sub-queries for serial execution | |
| CN107003906B (en) | Type-to-type analysis of cloud computing technology components | |
| US20170373932A1 (en) | Configuration discovery service data visualization | |
| US11016971B2 (en) | Splitting a time-range query into multiple sub-queries for parallel execution | |
| CN106598705A (en) | Asynchronous task scheduling method, device and system and electronic equipment | |
| CN115129782A (en) | Partition level connection method and device for distributed database | |
| CN104182295B (en) | A kind of data back up method and device | |
| CN113391913A (en) | Distributed scheduling method and device based on prediction | |
| Gopalakrishna et al. | Untangling cluster management with Helix | |
| CN108134810A (en) | A kind of method and its system of determining scheduling of resource component | |
| WO2016087640A1 (en) | Type-to-type analysis for cloud computing technical components | |
| CN116028217A (en) | Method and device for determining optimal path of data reading and electronic equipment | |
| CN117349284A (en) | Database splitting and table splitting method and device, electronic equipment and computer storage medium | |
| CN117455136A (en) | Automatic distribution method, system, software and computer equipment before PCB engineering | |
| CN117807025A (en) | Method, storage medium and equipment for synchronously operating data to time sequence library based on file | |
| CN114625655A (en) | Big data processing method and server applied to intelligent office |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180713 |
|
| RJ01 | Rejection of invention patent application after publication |