CN105740462A - Method for supporting data migration between different environments - Google Patents
Method for supporting data migration between different environments Download PDFInfo
- Publication number
- CN105740462A CN105740462A CN201610116440.2A CN201610116440A CN105740462A CN 105740462 A CN105740462 A CN 105740462A CN 201610116440 A CN201610116440 A CN 201610116440A CN 105740462 A CN105740462 A CN 105740462A
- Authority
- CN
- China
- Prior art keywords
- data
- etl
- database
- migration method
- java
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Abstract
The invention discloses a method for supporting data migration between different environments. The method includes the following steps of firstly, data extraction and storage, wherein to-be-migrated data of tables under a data source user is extracted in an ETL mode piece by piece according to the Java database connection technology and is temporarily stored in a Java stack memory; secondly, data operation and analysis, wherein the data temporarily stored in the memory is read according to the Java IO stream technology, according to the type of a database into which the data is to be written and the number of field columns of the tables, the data is converted into line records matched with the standard of the database, and the data is packaged into an available data object meeting ETL; thirdly, database writing-in of the data, wherein the standardized data is distributed to ETL multithread tasks according to the ETL database writing-in mechanism, continuous and rapid writing-in operation is conducted, and the data is inserted table by table till the line records are completely written to the tail. By means of the method, the data migration work can be conducted more efficiently and stably, faults can be reduced, and operation is easy.
Description
Technical field
The present invention relates to a kind of data migration method, particularly relate to a kind of data migration method supported between varying environment.
Background technology
Current internet has been enter into the epoch of data, in the face of growing mass data, it is necessary to be managed, test, analyze.Such as: data derive test library from storage facility located at processing plant, import certain a part of data from disparate databases, data analysis is done to new data base, inevitably need to be timed, quantitatively, have for Data Migration, how solving that the data efficient between varying environment migrates is emphasis.
Data are stored in different data bases, and such as: ORACLE, DB2, MYSQL etc., each data base it is also contemplated that concrete version.A such as scene, it is necessary to extract a part of data in these several data sources, import to new MYSQL storehouse and be analyzed, then complete this process, very time-consuming, effort, substantially increase the job costs of enterprise.
The Data Migration of current data base, major part is performed manually by, carry out data combing, verification by the dba personnel of specialty, then utilize export (swf) command to be operated, then import at target source, this can also use at same type of database manipulation, if but related to inter-library, the data importing a table of Cross-environment, or the other data volume of millions, whole process is extremely complex by what become, and difficulty.
Additionally, the mode of operation of this order, there is no daily record, also without the result data that interface is relevant, after having processed, it is not easy management and represents, also cannot verifying whether to meet result requirement, even occur that data are confusing, mistake such as repeats at the situation, the effective migration mechanism of neither one manages, and quality, efficiency all cannot be protected.
Therefore, the Data Transference Technology of existing database has a disadvantage in that 1) transport efficiency is low, every secondary entrance different pieces of information lab environment, performs loaded down with trivial details order；2) level requirement for treatment people is higher；3) do not keep track of, it is impossible to statistics.
Summary of the invention
The technical problem to be solved is to provide a kind of data migration method supported between varying environment, can allow the work of Data Migration, more smoothly run efficiently, reduce the generation of fault, and it is simple to operate, the problem being substantially reduced difficulty of starting with, ordinary person can be on duty through simple training.
The present invention solves that above-mentioned technical problem employed technical scheme comprise that a kind of data migration method supported between varying environment of offer, comprise the steps: a) data pick-up storage: utilize Java to connect the technology of data base, the data of data source user's following table to be migrated are extracted by ETL mode one by one, and is temporarily stored to the stack memory of Java；B) data operation analysis: utilize the IO Flow Technique of Java to read the data of interim internal memory, field number of columns according to the type of database to put in storage and table, convert data to mate the row record of this normalizing database, and be packaged into meet ETL can data object；C) data loading: utilize data base's writing mechanism of ETL, the data that specification is good are assigned to the task of ETL multithreading, carries out quickly write operation continuously, inserts data by table, until row record is completely written to end.
The above-mentioned data migration method between support varying environment, wherein, described data base is ORACLE, DB2 or MYSQL, and described step a) adopts multiple table data of multithreading operation support to extract simultaneously.
The above-mentioned data migration method between support varying environment, wherein, in described step c), ETL multithreading adopts queue to make to migrate data and buffers into storehouse, and it is 2～30,000 row/second that average data inserts speed.
The above-mentioned data migration method between support varying environment, wherein, described data migration method also includes arranging all-the-way tracking record, the data volume that statistics migrates, and adopts visualization display interface to carry out complete monitoring according to log recording.
The above-mentioned data migration method between support varying environment, wherein, described data migration method also includes arranging execution migration strategy with the form of task scheduling, and data source, target source and user right are configured management.
The present invention contrasts prior art following beneficial effect: the data migration method between support varying environment provided by the invention, stored by data pick-up, data operation analysis and data loading can allow the work of Data Migration, more smoothly run efficiently, reduce the generation of fault, and it is simple to operate, the problem being substantially reduced difficulty of starting with, ordinary person can be on duty through simple training.
Accompanying drawing explanation
Fig. 1 is the data mover system configuration diagram that the present invention supports between varying environment；
Fig. 2 is the Data Migration schematic flow sheet that the present invention supports between varying environment.
Detailed description of the invention
Below in conjunction with drawings and Examples, the invention will be further described.
Fig. 1 is the data mover system configuration diagram that the present invention supports between varying environment；Fig. 2 is the Data Migration schematic flow sheet that the present invention supports between varying environment.
Refer to Fig. 1 and Fig. 2, the data migration method between support varying environment provided by the invention, comprise the steps:
A) data pick-up storage: utilize Java to connect the technology of data base, the data of data source user's following table to be migrated are extracted by ETL (ExtractTransformLoad) mode one by one, and is temporarily stored to the stack memory of Java；
B) data operation analysis: utilize the IO Flow Technique of Java to read the data of interim internal memory, field number of columns according to the type of database to put in storage and table, convert data to mate the row record of this normalizing database, and be packaged into meet ETL can data object；
C) data loading: utilize data base's writing mechanism of ETL, the data that specification is good are assigned to the task of ETL multithreading, carries out quickly write operation continuously, inserts data by table, until row record is completely written to end.
The present invention data by specifying for data base, carry out quantitative extraction storage, operational analysis, include queue in, adopt buffer memory warehouse-in to insert；And with the format management migration strategy of task, the monitoring of whole process is done by the mode of journal displaying, efficiently solve the data migration problems between varying environment, attendant only needs to pass through interface operation, start to put in order and can complete work, the mode at complicated background commands interface is simplified operation, and the problem that start with difficulty is greatly reduced, ordinary person can be on duty through simple training.Core link is divided into following 3 layers:
1) data pick-up accumulation layer:
Mainly realize flow process: utilizing Java to connect the technology of data base, the data of certain data source user's following table are extracted by ETL mode one by one, and is temporarily stored to the stack memory of Java, this step completes data pick-up action.(this layer can support the different editions of the data bases such as ORACLE, DB2, MYSQL, supports that multiple table data extract simultaneously, supports that multithreading runs.The big data-handling capacity such as bill/inventory is on average in 0.4-1GB left and right per minute, and concrete processing speed is looked CPU, IO speed and network network speed and determined).
2) data operation analysis layer:
Mainly realize flow process: utilize the IO Flow Technique of Java to read the data of interim internal memory, field number of columns according to data base's (ORACLE, DB2, MYSQL) type to put in storage and table, convert data to mate the row record of this normalizing database, and be packaged into meet ETL can data object, this step completes data operation analysis action.
3) data efficient warehouse-in:
Mainly realizing flow process: utilize data base's writing mechanism of ETL, the data that specification is good are assigned to the task of ETL multithreading, carry out quickly write operation continuously, insert data by table, until row record is completely written to end, this step completes data efficient warehouse-in.(speed, on average up to 2～3w row/per second, supports disparate databases type, for instance ORACLE, DB2, MYSQL etc.)
In sum, particularly advantage of the invention is as follows: 1) greatly simplify specialty command operation difficulty, reduces attendant and requires threshold；2) the pattern striking order line, it has been advanced to visualized operation pattern；3) whole process has track record, it is possible to the data volume that statistics migrates；4) promote the transport efficiency of data, reduce cost of labor, reduce the fault because tired maloperation brings, indirectly decrease the loss of enterprise.
Although the present invention discloses as above with preferred embodiment; so it is not limited to the present invention, any those skilled in the art, without departing from the spirit and scope of the present invention; when doing a little amendment and perfect, therefore protection scope of the present invention is when with being as the criterion that claims define.
Claims (5)
1. the data migration method supported between varying environment, it is characterised in that comprise the steps:
A) data pick-up storage: utilize Java to connect the technology of data base, the data of data source user's following table to be migrated are extracted by ETL mode one by one, and is temporarily stored to the stack memory of Java；
B) data operation analysis: utilize the IO Flow Technique of Java to read the data of interim internal memory, field number of columns according to the type of database to put in storage and table, convert data to mate the row record of this normalizing database, and be packaged into meet ETL can data object；
C) data loading: utilize data base's writing mechanism of ETL, the data that specification is good are assigned to the task of ETL multithreading, carries out quickly write operation continuously, inserts data by table, until row record is completely written to end.
2. the data migration method supported between varying environment as claimed in claim 1, it is characterised in that described data base is ORACLE, DB2 or MYSQL, and described step a) adopts multiple table data of multithreading operation support to extract simultaneously.
3. the data migration method supported between varying environment as claimed in claim 1, it is characterised in that in described step c), ETL multithreading adopts queue to make to migrate data and buffers into storehouse, and it is 2～30,000 row/second that average data inserts speed.
4. the data migration method supported between varying environment as claimed in claim 1, it is characterized in that, described data migration method also includes arranging all-the-way tracking record, the data volume that statistics migrates, and adopts visualization display interface to carry out complete monitoring according to log recording.
5. the data migration method supported between varying environment as claimed in claim 1, it is characterised in that described data migration method also includes arranging execution migration strategy with the form of task scheduling, and data source, target source and user right are configured management.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610116440.2A CN105740462A (en) | 2016-03-02 | 2016-03-02 | Method for supporting data migration between different environments |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610116440.2A CN105740462A (en) | 2016-03-02 | 2016-03-02 | Method for supporting data migration between different environments |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105740462A true CN105740462A (en) | 2016-07-06 |
Family
ID=56248951
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610116440.2A Pending CN105740462A (en) | 2016-03-02 | 2016-03-02 | Method for supporting data migration between different environments |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105740462A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547892A (en) * | 2016-11-01 | 2017-03-29 | 山东浪潮云服务信息科技有限公司 | A kind of data resource management platform gathered based on internet data |
| CN106777345A (en) * | 2017-01-16 | 2017-05-31 | 山东浪潮商用系统有限公司 | A kind of data pick-up loading method based on mass data migration |
| CN107943889A (en) * | 2017-11-16 | 2018-04-20 | 郑州云海信息技术有限公司 | A kind of migration scheme of MySQL to KDB databases |
| CN108647270A (en) * | 2018-04-28 | 2018-10-12 | 尚谷科技（天津）有限公司 | A method of the Data Migration based on fault-tolerant time daily record |
| CN111367890A (en) * | 2020-03-13 | 2020-07-03 | 深圳壹账通智能科技有限公司 | Data migration method and device, computer equipment and readable storage medium |
| CN112597221A (en) * | 2020-12-17 | 2021-04-02 | 四川新网银行股份有限公司 | Test environment data extraction optimization execution method based on cross section data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739436A (en) * | 2009-09-28 | 2010-06-16 | 孙彬 | XML-based flexible data migration method |
| CN102156699A (en) * | 2010-02-11 | 2011-08-17 | 陈巍 | Data migration method based on JDOM revolving technology |
| US20120265726A1 (en) * | 2011-04-18 | 2012-10-18 | Infosys Limited | Automated data warehouse migration |
| CN102831211A (en) * | 2012-08-14 | 2012-12-19 | 中山大学 | Data sheet migration method based on sheet relation analysis |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN105095425A (en) * | 2015-07-17 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Cross-database transfer method and device for databases |
-
2016
- 2016-03-02 CN CN201610116440.2A patent/CN105740462A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739436A (en) * | 2009-09-28 | 2010-06-16 | 孙彬 | XML-based flexible data migration method |
| CN102156699A (en) * | 2010-02-11 | 2011-08-17 | 陈巍 | Data migration method based on JDOM revolving technology |
| US20120265726A1 (en) * | 2011-04-18 | 2012-10-18 | Infosys Limited | Automated data warehouse migration |
| CN102831211A (en) * | 2012-08-14 | 2012-12-19 | 中山大学 | Data sheet migration method based on sheet relation analysis |
| CN103440273A (en) * | 2013-08-06 | 2013-12-11 | 北京航空航天大学 | Data cross-platform migration method and device |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN105095425A (en) * | 2015-07-17 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Cross-database transfer method and device for databases |
Non-Patent Citations (1)
| Title |
|---|
| JEYSON的博客: "数据迁移工具kettle简单上手", 《HTTPS://BLOG.CSDN.NET/JS_SKY/ARTICLE/DETAILS/47833793?LOCATIONNUM=11》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106547892A (en) * | 2016-11-01 | 2017-03-29 | 山东浪潮云服务信息科技有限公司 | A kind of data resource management platform gathered based on internet data |
| CN106777345A (en) * | 2017-01-16 | 2017-05-31 | 山东浪潮商用系统有限公司 | A kind of data pick-up loading method based on mass data migration |
| CN106777345B (en) * | 2017-01-16 | 2020-07-28 | 浪潮软件科技有限公司 | Data extraction loading method based on mass data migration |
| CN107943889A (en) * | 2017-11-16 | 2018-04-20 | 郑州云海信息技术有限公司 | A kind of migration scheme of MySQL to KDB databases |
| CN108647270A (en) * | 2018-04-28 | 2018-10-12 | 尚谷科技（天津）有限公司 | A method of the Data Migration based on fault-tolerant time daily record |
| CN111367890A (en) * | 2020-03-13 | 2020-07-03 | 深圳壹账通智能科技有限公司 | Data migration method and device, computer equipment and readable storage medium |
| CN112597221A (en) * | 2020-12-17 | 2021-04-02 | 四川新网银行股份有限公司 | Test environment data extraction optimization execution method based on cross section data |
| CN112597221B (en) * | 2020-12-17 | 2023-04-11 | 四川新网银行股份有限公司 | Test environment data extraction optimization execution method based on cross section data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105740462A (en) | Method for supporting data migration between different environments | |
| US11714554B2 (en) | Method for aggregation optimization of time series data | |
| CN110297866A (en) | Method of data synchronization and data synchronization unit based on log analysis | |
| US20140101167A1 (en) | Creation of Inverted Index System, and Data Processing Method and Apparatus | |
| CN108694195B (en) | Management method and system of distributed data warehouse | |
| CN103294724B (en) | A kind of management method and system of database structure | |
| CN104239443B (en) | A kind of storage method of serialized data operation log | |
| CN102542071A (en) | Distributed data processing system and method | |
| CN104484131B (en) | The data processing equipment of multiple disks server and corresponding processing method | |
| CN109522312A (en) | A kind of data processing method, device, server and storage medium | |
| CN106126601A (en) | A kind of social security distributed preprocess method of big data and system | |
| US10311111B2 (en) | Search method and apparatus and storage medium | |
| CN103827852B (en) | Assemble WEB page on search engine results page | |
| CN102651008A (en) | Method and equipment for organizing data records in relational data base | |
| US11144424B2 (en) | System and method for recording the beginning and ending of job level activity in a mainframe computing environment | |
| CN103955530A (en) | Data reconstruction and optimization method of on-line repeating data deletion system | |
| CN104199963A (en) | Method and device for HBase data backup and recovery | |
| CN105630934A (en) | Data statistic method and system | |
| US20120317113A1 (en) | Computing device, storage medium, and method for processing bill of material of electronic products | |
| CN107665219A (en) | A kind of blog management method and device | |
| CA2950686C (en) | System and method for dynamic collection of system management data in a mainframe computing environment | |
| CN108959456A (en) | A kind of method of automatic recovery completion data | |
| CN111159117B (en) | Low-overhead file operation log acquisition method | |
| JP2017156931A (en) | Data storage device, data storage device control program, and data storage device control method | |
| CN109254961A (en) | A kind of distribution multi engine data quality management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Qingpu 201707 waiqingsong road Shanghai City, No. 588 Lane 7548 Building 1 R zone 1 room 113 Applicant after: Shanghai new torch network information technology Limited by Share Ltd Address before: Qingpu 201707 waiqingsong road Shanghai City, No. 588 Lane 7548 Building 1 R zone 1 room 113 Applicant before: SHANGHAI XINJU NETWORK INFORMATION TECHNOLOGY CO., LTD. |
|
| CB02 | Change of applicant information | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160706 |
|
| WD01 | Invention patent application deemed withdrawn after publication |