CN105528381A - Database data migration method and system - Google Patents
Database data migration method and system Download PDFInfo
- Publication number
- CN105528381A CN105528381A CN201410583273.3A CN201410583273A CN105528381A CN 105528381 A CN105528381 A CN 105528381A CN 201410583273 A CN201410583273 A CN 201410583273A CN 105528381 A CN105528381 A CN 105528381A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- migration
- task
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 42
- 238000013508 migration Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 24
- 230000008569 process Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims 4
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 239000011159 matrix material Substances 0.000 claims 1
- 230000007704 transition Effects 0.000 description 4
- 230000008520 organization Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 230000001617 migratory effect Effects 0.000 description 2
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a database data migration method which is applied to a distributed-system cluster environment and is used for migrating data between a first database and a second database. The method includes the steps of configuring table task information corresponding to multiple table tasks of a database migration task, wherein the table tasks can be dispatched in batch; reading data of to-be-migrated source data tables of the table tasks from the first database according to the dispatched task tables, subjecting the data of the source data tables to sharding to acquire multiple sharding data tables, and importing the sharding data tables into a distributed file system; reading the sharding data tables from the distributed file system, and exporting the sharding data tables into the second database. According to the migration method, different data can be migrated from one database to another database only through one-time configuration, so that speed and stability of data migration are increased. The invention further provides a database data migration system.
Description
Technical field
The application relates to the method for Data Migration between disparate databases in computer technology, particularly relates to the method and system of Data Transferring among Different Structure Database.
Background technology
In early stage internet, applications, bottom data generally takes the storage scheme of all data of single library storage.Along with the development of internet, applications, the surge of customer volume, datum number storage amount exponentially increases progressively, and the restriction of bottom data list library storage scheme even limits further expanding of internet, applications.For this reason, for solving the bottleneck problem of single library storage, another need be progressively adopted to support the solution laterally stored.And the bottleneck of data storage will be solved, existing storage organization must be switched, but because current bottom data amount is very huge, while switching storage organization, how original mass data being moved on new support storage organization extending transversely is also a very large bottleneck point.
The existing utility carrying out means that in internet, Data Migration is used or provided by disparate databases by the statistical conversion of former database in file, again these data files are imported to new tables of data by another data base tool, or write in disparate databases and apply relevant program, after in a program data being checked out by query statement from former database, by program, data are inserted into new database again, or utilize cloud Data Migration Tools, by the distributed file system (HadoopDistributedFileSystem of the data importing in a relevant database to distributed system cluster Hadoop, HDFS) in, also can by the data importing of HDFS in relevant database.By Sqoop, on the basis based on Hadoop distributed treatment, data can be fetched in HDFS from former database, then data are taken out from HDFS import in new database.Find out that the core of the Data Migration Tools in current internet is all data to derive from former database thus, again by data importing in new database, these technical schemes have the following disadvantages, and are first cannot carry out batch operation in multilist mass data situation; Secondly, data volume cannot be avoided to exceed the problem of server handling ability; Again, existing database data migration instrument generally all cannot support the secondary treating to data, namely all can not support point storehouse migration to former database data; Finally, a lot of verifying function not after supported data migration of the Data Migration Tools in current internet.
Summary of the invention
In view of this, be necessary to provide a kind of database data migration method and system, to solve the problem of the speed, stability and the data correctness that exist in existing database Data Migration.
The application provides a kind of database data migration method, is applied in distributed system cluster environment, and for migration data between the first database and the second database, the method comprises:
The table mission bit stream corresponding by multiple table tasks of lot size scheduling of configuration database migration task, source data table corresponding with this table task in the first database is carried out Data Migration by described each table task;
The data needing the source data table moved are read this table task from the first database;
And a point storehouse is carried out to the data of this source data table obtain multiple points of database data tables, more the plurality of point of database data table is imported in distributed file system; And from described distributed file system, the plurality of point of database data table is exported in the second database.
Further, described moving method also comprises: compare the data in the first database and the second database, verify data in transition process whether have disappearance and data whether imperfect.
The application also provides a kind of database data migration system, operates in distributed system cluster environment, and for migration data between the first database and the second database, this migratory system comprises:
Configuration module, for the table mission bit stream corresponding to multiple table tasks of configuration database migration task, described each table task refers to the migration task of source data table in the first database being carried out to Data Migration;
Dispatching control module, can read described table mission bit stream, and multiple table task described in lot size scheduling;
Data importing module, for reading this table task from the first database the data needing the source data table moved according to dispatched table task, and a point storehouse is carried out to the data of this source data table obtain multiple points of database data tables, more the plurality of point of database data table is imported in distributed file system; And
Statistical conversion module, reads described multiple points of database data tables, then is exported in the second database by the plurality of point of database data table from described distributed file system.
Further, described migratory system also comprises data check module, this data check module for comparing the data in the first database and the second database, verify data in transition process whether have disappearance and data whether imperfect.
Compared with prior art, each tables of data is used as a migration task by the application's database data migration method and system, using the Main Means that database utility or custom program read and write data as heterogeneous database, these tasks are run in distributed type assemblies, add self-defining point of storehouse algorithm in data handling simultaneously, by read Data Placement in multiple file, then in task calling data storehouse utility or custom program by data importing in multiple new database.Because each table is a migration task, the problem in different table different pieces of information source can be had and divide clearly, can support that multiple task batch carries out simultaneously and do not interact, accelerate the speed of Data Migration greatly.After migration completes, the data between Xin Ku and old storehouse can be verified, find out inconsistent data.Therefore the method for the process data exporting that the application is relatively traditional, improves the speed of migration, provides data check function, ensure that the stability in transition process and the data correctness in transition process.
Claims (4)
1. a database data migration method, be applied in distributed system cluster environment, for migration data between the first database and the second database, it is characterized in that, the method comprises: the table mission bit stream corresponding by multiple table tasks of lot size scheduling of configuration database migration task, and source data table corresponding with this table task in the first database is carried out Data Migration by described each table task; The data needing the source data table moved are read this table task from the first database; And a point storehouse is carried out to the data of this source data table obtain multiple points of database data tables, more the plurality of point of database data table is imported in distributed file system; And from described distributed file system, the plurality of point of database data table is exported in the second database.
2. database data migration method as claimed in claim 1, is characterized in that, configures table mission bit stream corresponding to multiple table task by table configuration file.
3. database data migration method as claimed in claim 2, it is characterized in that, corresponding each table task, described table configuration file comprises source configuration and target configuration, the configuration of described source gives in the first database the information needing the source data table moved, and described target configuration gives will the information of the target matrix in the second database needing the source data table of migration to move to.
4. database data migration method as claimed in claim 1, it is characterized in that, data importing in first database comprised to the process in distributed file system: obtain and load the table mission bit stream of dispatched table task, described table mission bit stream comprises this table task to the cutting field of the source data table in requisition for migration and point storehouse field information; According to multiple sections that the cutting field in described table mission bit stream will need the source data table of migration to be divided into the first quantity, and ask the corresponding each section formation of distributed system cluster first mapping tasks, described each first mapping tasks is with the first DataBase combining and read of source data table and cut into slices; Further request distributed system cluster is in described each first mapping tasks, carry out to each section that this first mapping tasks reads point database data table that point storehouse obtains the second quantity according to point storehouse field in described table mission bit stream, request distributed system cluster forms a first abbreviation task according to each point of database data table correspondence obtained behind described point of storehouse further; And by each first abbreviation task, the data of each point of database data table obtained behind point storehouse are written in distributed file system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410583273.3A CN105528381A (en) | 2014-10-27 | 2014-10-27 | Database data migration method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410583273.3A CN105528381A (en) | 2014-10-27 | 2014-10-27 | Database data migration method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105528381A true CN105528381A (en) | 2016-04-27 |
Family
ID=55770607
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410583273.3A Pending CN105528381A (en) | 2014-10-27 | 2014-10-27 | Database data migration method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105528381A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956191A (en) * | 2016-06-13 | 2016-09-21 | 浪潮(北京)电子信息产业有限公司 | Data migration method and system |
| CN107480224A (en) * | 2017-09-11 | 2017-12-15 | 爱普（福建）科技有限公司 | The configuration data of control station realizes the device of data sharing with third party database |
| CN107958057A (en) * | 2017-11-29 | 2018-04-24 | 苏宁云商集团股份有限公司 | A kind of code generating method and device for being used for Data Migration in heterogeneous database |
| CN108241632A (en) * | 2016-12-23 | 2018-07-03 | 航天星图科技（北京）有限公司 | A kind of data verification method of data base-oriented Data Migration |
| CN109144977A (en) * | 2018-08-14 | 2019-01-04 | 五八有限公司 | A kind of data migration method, device, equipment and storage medium |
| CN113204538A (en) * | 2021-04-27 | 2021-08-03 | 北京百度网讯科技有限公司 | Method, apparatus, device, medium and program product for data migration |
-
2014
- 2014-10-27 CN CN201410583273.3A patent/CN105528381A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105956191A (en) * | 2016-06-13 | 2016-09-21 | 浪潮(北京)电子信息产业有限公司 | Data migration method and system |
| CN108241632A (en) * | 2016-12-23 | 2018-07-03 | 航天星图科技（北京）有限公司 | A kind of data verification method of data base-oriented Data Migration |
| CN108241632B (en) * | 2016-12-23 | 2022-01-14 | 中科星图股份有限公司 | Data verification method oriented to database data migration |
| CN107480224A (en) * | 2017-09-11 | 2017-12-15 | 爱普（福建）科技有限公司 | The configuration data of control station realizes the device of data sharing with third party database |
| CN107958057A (en) * | 2017-11-29 | 2018-04-24 | 苏宁云商集团股份有限公司 | A kind of code generating method and device for being used for Data Migration in heterogeneous database |
| CN107958057B (en) * | 2017-11-29 | 2022-04-05 | 苏宁易购集团股份有限公司 | Code generation method and device for data migration in heterogeneous database |
| CN109144977A (en) * | 2018-08-14 | 2019-01-04 | 五八有限公司 | A kind of data migration method, device, equipment and storage medium |
| CN113204538A (en) * | 2021-04-27 | 2021-08-03 | 北京百度网讯科技有限公司 | Method, apparatus, device, medium and program product for data migration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103793424B (en) | database data migration method and system | |
| CN105528381A (en) | Database data migration method and system | |
| US10628449B2 (en) | Method and apparatus for processing database data in distributed database system | |
| US10872066B2 (en) | Systems and methods of database tenant migration | |
| US9424274B2 (en) | Management of intermediate data spills during the shuffle phase of a map-reduce job | |
| US8738650B2 (en) | Distributed processing of streaming data records | |
| CN102968498A (en) | Method and device for processing data | |
| CN108241632B (en) | Data verification method oriented to database data migration | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| CN110209728A (en) | A kind of Distributed Heterogeneous Database synchronous method, electronic equipment and storage medium | |
| CN105956666B (en) | A kind of machine learning method and system | |
| CN104112008A (en) | Multi-table data association inquiry optimizing method and device | |
| CN106919697B (en) | Method for simultaneously importing data into multiple Hadoop assemblies | |
| CN104111936A (en) | Method and system for querying data | |
| CN105630778A (en) | DB data migration method and system | |
| CN104915414A (en) | Data extraction method and device | |
| CN105447172A (en) | Data processing method and system under Hadoop platform | |
| CN106708902A (en) | Database data migration method and system | |
| WO2016101751A1 (en) | Master and slave balancing method and device in distributed storage system | |
| CN104298761A (en) | Implementation method for master data matching between heterogeneous software systems | |
| CN107798120B (en) | Data conversion method and device | |
| CN102521304A (en) | Hash based clustered table storage method | |
| CN101645073A (en) | Method for guiding prior database file into embedded type database | |
| US9239852B1 (en) | Item collections | |
| CN116662019B (en) | Request distribution method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160427 |
|
| WD01 | Invention patent application deemed withdrawn after publication |