CN117312272A - Data migration solution method and system based on multi-source heterogeneous data scene - Google Patents
Data migration solution method and system based on multi-source heterogeneous data scene Download PDFInfo
- Publication number
- CN117312272A CN117312272A CN202311264750.5A CN202311264750A CN117312272A CN 117312272 A CN117312272 A CN 117312272A CN 202311264750 A CN202311264750 A CN 202311264750A CN 117312272 A CN117312272 A CN 117312272A
- Authority
- CN
- China
- Prior art keywords
- data
- source heterogeneous
- migrated
- database
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 35
- 230000005012 migration Effects 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000013507 mapping Methods 0.000 claims abstract description 17
- 238000013506 data mapping Methods 0.000 claims abstract description 7
- 238000013499 data model Methods 0.000 claims abstract description 7
- 230000008676 import Effects 0.000 claims description 7
- 230000003068 static effect Effects 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000003780 insertion Methods 0.000 claims description 5
- 230000037431 insertion Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 230000004048 modification Effects 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 abstract description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration solution method and a system based on a multi-source heterogeneous data scene, and relates to the technical field of data migration. The data migration solution method based on the multi-source heterogeneous data scene comprises the following steps: step A: acquiring a data range to be migrated according to the current time node; and (B) step (B): creating a temporary intermediate table for storing stock data and an index of the temporary intermediate table, and checking accuracy and completeness; step C: the DB2 data is flatly migrated to a isomorphic database table under TDSQL, and data mapping among different data models is realized in an SQL query mapping mode; step E: verifying whether the data quantity and the summarized value recorded by the migrated library table are consistent with the stock data, and generating a detail comparison report. The invention completes the construction of a basic data migration flow, and a researcher can realize the data migration of the new and old system multi-source heterogeneous data scene only by completing the templated query mapping SQL between the new and old system data tables.
Description
Technical Field
The invention relates to the technical field of data migration, in particular to a data migration solution method and system based on a multi-source heterogeneous data scene.
Background
When switching, reconstructing and upgrading new and old systems, the stock data is required to be migrated from the old system service architecture to the new system so as to maintain the continuity of the service. And this migration action may rely on the form of sql query insertion or file importation. However, when the service architecture of the new and old systems is inconsistent, the database selection of the new and old systems is inconsistent, and other complex scenes, a single migration means cannot meet the migration requirement of the complex scenes.
Disclosure of Invention
(one) solving the technical problems
Aiming at the defects of the prior art, the invention provides a data migration solution method and a system based on a multi-source heterogeneous data scene, which solve the problem that a single migration means cannot meet the migration requirement of a complex scene.
(II) technical scheme
In order to achieve the above purpose, the invention is realized by the following technical scheme: a data migration solution method and system based on a multi-source heterogeneous data scene comprises the following steps:
step A: acquiring a data range to be migrated according to the current time node;
and (B) step (B): creating a temporary intermediate table for storing stock data and an index of the temporary intermediate table, and checking accuracy and completeness;
step C: the DB2 data is flatly migrated to a isomorphic database table under TDSQL, and data mapping among different data models is realized in an SQL query mapping mode;
step E: verifying whether the data quantity and the summarized value recorded by the migrated library table are consistent with the stock data, and generating a detail comparison report.
Preferably, in the step C, the method of updating the standard table with poor insertion efficiency is adopted to insert the non-standard table and then export the file to import the standard table.
Preferably, in the step C, a mapping relationship between the data record adopted for the database table and the data virtual partition is determined according to the data virtual partition rule;
and mapping the data records in the database table to the corresponding data virtual partitions according to the column values of the database table, the total number of the data virtual partitions and the mapping relation between the adopted data records and the data virtual partitions.
Preferably, the step a needs to start the server and complete registration between the first execution, and the temporary intermediate table and the index of the temporary intermediate table are created by the server.
Preferably, the data information in the temporary intermediate table includes a name of the data, a byte size of the data, and a modification time of the data.
Preferably, the data information in the temporary intermediate table includes a name of the data, a byte size of the data, and a modification time of the data.
Preferably, in the step E, if at least one of the data amount, the total value and the stock data recorded in the library table is consistent, a fault report is generated, and whether to re-transfer the data is selected.
A data migration system based on a multi-source heterogeneous data scenario, comprising:
multiple environment support module: the unified configuration of the database connection is realized aiming at the database connection with different multiple environments;
static parameter acquisition module: acquiring a data range to be migrated according to the current time node;
a temporary table creation module: creating a temporary intermediate table for storing stock data;
index creation and verification module: creating an index of the temporary intermediate table and checking accuracy and completeness;
and a data flattening module: solving database isomerism, namely flatly moving DB2 data to a database table isomorphic under TDSQL;
and a data deformation module: solving data mapping among different data models;
large table data export import module: for the standard table with poor updating and inserting efficiency, inserting a non-standard table and then exporting a file to import into the standard table;
static migration checking module: and verifying whether the data quantity and the summarized value of the migrated library table record are consistent with the stock data.
And a service verification module: and verifying whether the stock service data are consistent or not, and generating a detail comparison report.
Working principle: the invention designs a data file for acquiring the table structure of the old system database from the DB2 database, and synchronizes the data in the data file to the TDSQL database, thereby realizing the data aggregation of the multi-source data scene. According to the method, the data transformation is realized in a SQL query mapping mode according to the synchronous data of the old system database table structure in the TDSQL, the stock data is queried in a related manner according to the mapping relation, the stock data of the new system data structure is obtained, and the stock data is inserted into the database of the new system, so that the data synchronization of heterogeneous scenes is solved.
(III) beneficial effects
The invention provides a data migration solution method and a system based on a multi-source heterogeneous data scene. The beneficial effects are as follows: the invention completes the construction of a basic data migration flow, and research and development personnel can realize the data migration of the new and old system multi-source heterogeneous data scene only by completing the templated query mapping SQL between the new and old system data tables, thereby meeting the migration requirement in the complex scene and greatly simplifying the migration process.
Drawings
FIG. 1 is a schematic flow chart of the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Embodiment one:
as shown in fig. 1, an embodiment of the present invention provides a data migration solution method based on a multi-source heterogeneous data scenario, including the following steps:
step A: and acquiring a data range to be migrated according to the current time node, starting a server and finishing registration between the first execution, wherein the temporary intermediate table and the index of the temporary intermediate table are created through the server.
And (B) step (B): creating a temporary intermediate table for storing the stock data and an index of the temporary intermediate table, checking to be accurate and complete, and combining the data with the same data information, wherein the data information in the temporary intermediate table comprises the name of the data, the byte size of the data and the modification time of the data.
Step C: the DB2 data is flatly migrated to a isomorphic database table under the TDSQL, data mapping among different data models is realized in an SQL query mapping mode, and for updating a standard table with poor insertion efficiency, a method of inserting a non-standard table first and then exporting a file to be imported to the standard table is adopted for processing, and the mapping relation between data records adopted for the database table and data virtual partitions is determined according to the data virtual partition rules; and mapping the data records in the database table to the corresponding data virtual partitions according to the column values of the database table, the total number of the data virtual partitions and the mapping relation between the adopted data records and the data virtual partitions.
The TDSQL is an autonomous controllable high-consistency distributed database product compatible with MySQL and introduced by vacation. Here we emphasize a point that it is highly compatible with mysql—tdsql is fully compatible with MySQL protocol, and is fully autonomous and controllable, data strong consistency. And the second is that the TDSQL has the distributed property and has a flexible expansion and high availability architecture. In the internet industry, massive user traffic scenes are very common, and if a database does not have scalability, it is difficult to cope with the following situations: large-scale sales promotion of electronic commerce, and sudden increase flow scenes such as spring festival rushing to red packets, which are all the practical tests for coping with mass user flow by the database.
SQL (Structured Query Language) is a database language with multiple functions such as data manipulation and data definition, and the language has the characteristic of interactivity, so that great convenience can be provided for users, and the database management system should make full use of the SQL language to improve the working quality and efficiency of the computer application system. The SQL language can be independently applied to the terminal, and can also be used as a sub-language to provide effective assistance for other programming.
Step E: verifying whether the data quantity and the summary value recorded by the migrated library table are consistent with the stock data or not, generating a detail comparison report, generating a fault report if at least one of the data quantity and the summary value recorded by the library table is consistent with the stock data, and selecting whether to transfer the data again or not.
A data migration system based on a multi-source heterogeneous data scenario, comprising:
multiple environment support module: the unified configuration of the database connection is realized aiming at the database connection with different environments.
Static parameter acquisition module: and acquiring a data range needing to be migrated according to the current time node.
A temporary table creation module: a temporary intermediate table is created that stores inventory data.
Index creation and verification module: an index of the temporary intermediate table is created and verified to be accurate and complete.
And a data flattening module: and (3) solving database isomerism, and flattening DB2 data to a database table isomorphic under TDSQL.
And a data deformation module: data mapping between different data models is solved.
Large table data export import module: for the update of the standard table with poor insertion efficiency, the standard table is inserted first, and then the export file is imported into the standard table.
Static migration checking module: and verifying whether the data quantity and the summarized value of the migrated library table record are consistent with the stock data.
And a service verification module: and verifying whether the stock service data are consistent or not, and generating a detail comparison report.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. A data migration solution method based on a multi-source heterogeneous data scene, comprising the steps of:
step A: acquiring a data range to be migrated according to the current time node;
and (B) step (B): creating a temporary intermediate table for storing stock data and an index of the temporary intermediate table, and checking accuracy and completeness;
step C: the DB2 data is flatly migrated to a isomorphic database table under TDSQL, and data mapping among different data models is realized in an SQL query mapping mode;
step E: verifying whether the data quantity and the summarized value recorded by the migrated library table are consistent with the stock data, and generating a detail comparison report.
2. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: in the step C, a method of inserting a non-card table and then exporting a file to import the card table is adopted for updating the card table with poor insertion efficiency.
3. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: c, determining a mapping relation between the data record adopted for the database table and the data virtual partition according to the data virtual partition rule;
and mapping the data records in the database table to the corresponding data virtual partitions according to the column values of the database table, the total number of the data virtual partitions and the mapping relation between the adopted data records and the data virtual partitions.
4. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: step A needs to start the server and complete registration between the first execution, and the temporary intermediate table and the index of the temporary intermediate table are created by the server.
5. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: the data information in the temporary intermediate table includes the name of the data, the byte size of the data, and the modification time of the data.
6. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: and B, combining the data with the same data information.
7. The method for solving the data migration problem based on the multi-source heterogeneous data scene according to claim 1, wherein: and E, if at least one item of data quantity, total value and stock data recorded by the library table is consistent, generating a fault report, and selecting whether to transfer the data again.
8. A data migration system based on a multi-source heterogeneous data scenario, comprising:
multiple environment support module: the unified configuration of the database connection is realized aiming at the database connection with different multiple environments;
static parameter acquisition module: acquiring a data range to be migrated according to the current time node;
a temporary table creation module: creating a temporary intermediate table for storing stock data;
index creation and verification module: creating an index of the temporary intermediate table and checking accuracy and completeness;
and a data flattening module: solving database isomerism, namely flatly moving DB2 data to a database table isomorphic under TDSQL;
and a data deformation module: solving data mapping among different data models;
large table data export import module: for the standard table with poor updating and inserting efficiency, inserting a non-standard table and then exporting a file to import into the standard table;
static migration checking module: verifying whether the data quantity and the summarized value recorded by the migrated library table are consistent with the stock data;
and a service verification module: and verifying whether the stock service data are consistent or not, and generating a detail comparison report.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311264750.5A CN117312272A (en) | 2023-09-27 | 2023-09-27 | Data migration solution method and system based on multi-source heterogeneous data scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311264750.5A CN117312272A (en) | 2023-09-27 | 2023-09-27 | Data migration solution method and system based on multi-source heterogeneous data scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117312272A true CN117312272A (en) | 2023-12-29 |
Family
ID=89254847
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311264750.5A Pending CN117312272A (en) | 2023-09-27 | 2023-09-27 | Data migration solution method and system based on multi-source heterogeneous data scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117312272A (en) |
-
2023
- 2023-09-27 CN CN202311264750.5A patent/CN117312272A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101504672B (en) | Method and system for dynamically configuring physical data table | |
| CN108399256A (en) | Heterogeneous database content synchronization method, device and middleware | |
| US20120203745A1 (en) | System and method for range search over distributive storage systems | |
| CN104572689A (en) | Data synchronizing method, device and system | |
| CN112380180A (en) | Data synchronization processing method, device, equipment and storage medium | |
| CN106934048A (en) | Online data moving method, agent node | |
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| US7624117B2 (en) | Complex data assembly identifier thesaurus | |
| CN105405070A (en) | Distributed memory power grid system construction method | |
| CN112988702A (en) | Heterogeneous data source real-time data transmission method and system, storage medium and terminal | |
| CN102355499A (en) | Cloud computing system | |
| CN115470284A (en) | Method and device for importing multi-source heterogeneous data source into Janusgraph database | |
| CN109388659A (en) | Date storage method, device and computer readable storage medium | |
| CN108959391B (en) | Apparatus, system, method, and storage medium for presenting database cluster architecture diagram | |
| CN111914028A (en) | Method and device for synchronizing data relation of heterogeneous data sources based on graph increment | |
| CN117312272A (en) | Data migration solution method and system based on multi-source heterogeneous data scene | |
| CN112416944A (en) | Method and equipment for synchronizing service data | |
| CN114925075B (en) | Real-time dynamic fusion method for multi-source time-space monitoring information | |
| CN110532243A (en) | Data processing method, device and electronic equipment | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| US7536398B2 (en) | On-line organization of data sets | |
| CN113886505B (en) | Management system for realizing dynamic modeling based on search engine and relational database | |
| CN115587141A (en) | Database synchronization method and device | |
| CN114168122A (en) | Data script generation method and device | |
| CN108595488B (en) | Data migration method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |