CN117971809A - Data migration method and system based on heterogeneous database - Google Patents
Data migration method and system based on heterogeneous database Download PDFInfo
- Publication number
- CN117971809A CN117971809A CN202410308033.6A CN202410308033A CN117971809A CN 117971809 A CN117971809 A CN 117971809A CN 202410308033 A CN202410308033 A CN 202410308033A CN 117971809 A CN117971809 A CN 117971809A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- field
- migration
- fields
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 title claims abstract description 68
- 238000013508 migration Methods 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000011161 development Methods 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 13
- 230000004048 modification Effects 0.000 claims description 15
- 238000012986 modification Methods 0.000 claims description 15
- 238000007726 management method Methods 0.000 claims description 14
- 238000013524 data verification Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 238000012360 testing method Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 3
- 238000007670 refining Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000008520 organization Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013502 data validation Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data migration, in particular to a data migration method and system based on a heterogeneous database, comprising the following steps: processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment; importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file; and verifying the migrated data. The invention ensures that the original data is migrated to the new database more efficiently and accurately, and the field customization processing can be performed according to the requirements.
Description
Technical Field
The invention relates to the technical field of data migration, in particular to a data migration method and system based on a heterogeneous database.
Background
The database management system which is independently researched and developed or introduced by enterprises or institutions and is subjected to localized transformation is usually optimized according to the requirements and characteristics of markets, the localized transformation is carried out according to the specific requirements of the enterprises or institutions, the database management system can be better adapted to the local business environment and specific data processing requirements, the database management system can be customized according to the actual requirements of the enterprises or institutions, the cost of purchasing an expensive external database management system is avoided, the data management requirements of the enterprises or institutions can be better met, and the efficiency of data processing and management is improved.
However, when an enterprise or an organization is used to independently develop or introduce and locally reform a database management system, original data resources cannot be lost, and a small amount of adjustment is also provided for a data structure when the software is locally developed, so that the database migration of the database management system which is independently developed or introduced and locally reform by the enterprise or the organization is a key link of the software local development and is also a difficulty of the software local development.
Disclosure of Invention
The invention provides a data migration method and system based on a heterogeneous database, which aims to solve the existing problems: after an enterprise or an organization autonomously develops or introduces and locally reforms a database management system, the problem of migration of original data to a new database is solved.
The invention discloses a data migration method and a system based on a heterogeneous database, which adopts the following technical scheme:
One embodiment of the invention provides a data migration method based on heterogeneous databases, which comprises the following steps:
processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment;
importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file;
And verifying the migrated data.
Further, the method for processing the data in the database to be migrated and generating the database object, and modifying the table structure in the new database according to the service variation or specification to obtain the database development environment comprises the following specific steps:
specifically, migrating the database table, and recording the relevant authorities of objects such as views, functions, triggers, synonyms and the like; implementing the method and the trigger by codes, using a Restful interface for external views, modifying a table structure according to service variation or specification problems, exporting the table and the views to generate SQL, combing, and generating database objects in a new database according to the SQL; the database is developed, the method and the trigger are realized by codes, the external view is realized by using a Restful interface, and the structure of the table is modified according to service variation or specification problems in the development process, so that the database development environment is obtained.
Further, the data is configured according to a migration tool, and the specific method comprises the following steps:
an automatic corresponding stage, wherein when the table names of the source database and the target database are the same, the data is automatically written into the corresponding table;
the default level of the system, the management system unifies the added fields or unifies the fields needing to be refreshed, and refreshes all the tables containing the fields;
The database object level is specially configured for specific table modification and specific field modification.
Further, when the table names and the field names of the source database and the target database are the same, the data is automatically written into the corresponding table, and the specific method comprises the following steps:
inquiring table data from a source database, and when the data volume is too large, creating an intermediate table or carrying out paging inquiry and putting the intermediate table into a memory; inserting the data queried in the previous step into a target database according to the homonym table and the homonym; if there is paging, it is necessary to cyclically insert all the data in the table.
Further, the management system unifies added fields or unifies fields to be refreshed, and refreshes all tables containing the fields, and the specific method comprises the following steps:
And (3) traversing all fields in the lookup table to query system-level configuration, wherein a table field containing the configuration selects a refreshing field according to a configuration rule or fills the redundant field with a value according to a value of a certain column in the table and table data of a source database.
Further, the specific configuration for specific table modification and specific field modification comprises the following specific methods:
The configuration is refreshed to the database, defaulting or modifying existing values for specific fields of a specific table.
Further, the verification of the migrated data comprises the following specific methods:
inquiring all target data and the main key thereof, taking out each piece of data, calculating CRC value, inquiring source data according to the main key, modifying the data according to configuration, comparing the data, storing the data which are failed to be compared into a test report, and finally refining inconsistent data in the test report into fields.
Further, the specific method for selecting the refresh field according to the configuration rule includes:
Refreshing a field so that the value of the field in each table is a certain value; if there is this field in the table, the value of this field needs to be modified all the way to the value of this field, and then the "if all updated" parameter is configured.
Further, the method for filling the redundant field with the value according to the value of a certain column in the table and the table data of the source database comprises the following specific steps:
If the table contains a certain field and other fields containing the field, the field values of the other fields containing the field need to be queried from the base table according to the field values, and the update target table is filled in.
The embodiment of the invention provides a data migration system based on a heterogeneous database, which comprises an environment configuration module, a data migration module and a data verification module, wherein:
The environment configuration module is used for processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment;
The data migration module is used for importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file;
and the data verification module is used for verifying the migrated data.
The technical scheme of the invention has the beneficial effects that: the invention solves the problem of incompatibility of the formats of data migration in the process of localized development of software, ensures the consistency and the integrity of data, avoids the problem of data loss or damage, and can carry out special configuration on specific table and field modification according to the requirements so as to realize the customization treatment of specific fields.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of steps of a heterogeneous database-based data migration method of the present invention;
FIG. 2 is a block diagram of a heterogeneous database-based data migration system according to the present invention;
FIG. 3 is a migration diagram of a heterogeneous database according to the present invention;
FIG. 4 is a flow chart of a migration tool based on heterogeneous databases according to the present invention.
Detailed Description
In order to further describe the technical means and effects adopted by the present invention to achieve the preset purposes, the following detailed description refers to the specific implementation, structure, characteristics and effects of a data migration method and system based on heterogeneous databases according to the present invention with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" means that the embodiments are not necessarily the same. Furthermore, the particular features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
The following specifically describes a specific scheme of a data migration method and system based on heterogeneous databases provided by the invention with reference to the accompanying drawings.
Referring to fig. 1, a flowchart illustrating a data migration method based on heterogeneous databases according to an embodiment of the present invention is shown, the method includes the following steps:
Step S001: and processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment.
Specifically, migrating the database table, and recording the relevant authorities of objects such as views, functions, triggers, synonyms and the like; implementing the method and the trigger by codes, using a Restful interface for external views, modifying a table structure according to service variation or specification problems, exporting the table and the views to generate SQL, combing, and generating database objects in a new database according to the SQL; the database is developed, the method and the trigger are realized by codes, the external view uses a Restful interface, and the table structure is modified possibly due to service variation or specification problems in the development process, so that the database development environment is obtained.
Step S002: the data is migrated according to the data migration tool configuration and the corresponding migration configuration file by importing the database object,
Specifically, the migration database object is imported from the acquired development environment, and the data is migrated according to the migration tool configuration and the corresponding migration configuration file.
After the migration database object is imported from the development environment, the data migration tool needs to be configured, and the configuration of the migration tool is mainly divided into an automatic corresponding level, a system default level and a database object level.
Specifically, the automatic corresponding stage is realized, and when the table names and field names of the source database and the target database are the same, the data is automatically written into the corresponding table; the default level of the system, the management system unifies the added fields or unifies the fields needing to be refreshed, and refreshes all the tables containing the fields; the database object level is specially configured for specific table modification and specific field modification.
Step S003: and when the table names of the source database and the target database are the same, the data is automatically written into the corresponding table.
Specifically, the data of the table is inquired from the source database, and when the data volume is too large, an intermediate table is required to be created or paging inquiry is carried out and the intermediate table is put into a memory; inserting the data queried in the previous step into a target database according to the homonym table and the homonym; if there is paging, it is necessary to cyclically insert all the data in the table.
Step S004: the management system unifies the added fields or unifies the fields that need to be refreshed, and refreshes all the tables containing these fields.
Specifically, all fields in the table are searched, the system level configuration is traversed, and the table field containing the configuration is used for selecting a refreshing field according to the configuration rule or filling the redundant field with a value according to the value of a certain column in the table and the table data of the source database.
Refresh field: refreshing a field so that the value of the field in each table is a certain value; if there is this field in the table, the value of this field needs to be modified all the way to the value of this field, and then the "if all updated" parameter is configured.
Filling the redundant field with a value according to the value of a certain column in the table and the table data of the source database: if the table contains a certain field and other fields containing the field, the field values of the other fields containing the field need to be queried from the base table according to the field values, and the update target table is filled in.
Step S005: special configurations are made for specific table modifications, specific field modifications.
Specifically, the configuration is refreshed to the database, defaulting or modifying existing values for specific fields of a specific table.
Step S006: and verifying the migrated data.
It should be noted that, the meaning of performing data verification after data migration is to ensure the integrity, accuracy and consistency of the data in the migration process. The data can be found out to be lost, damaged or wrong possibly occurring in the migration process through data verification, and the data can be repaired and corrected in time, so that the quality and the reliability of the data are guaranteed. At the same time, data validation can also help to confirm whether the migrated data conforms to the expected structure and format, and whether the business requirements and prescribed criteria are met. This avoids potential data quality problems and reduces the risk and impact that may occur after data migration. Therefore, data verification is a very important ring in the data migration process, and has important significance for guaranteeing the integrity and reliability of data.
Specifically, verifying the correctness of the data includes verifying the integrity of the table structure, the integrity of the data, and the number of bits reserved by the value. The specific data verification method comprises the following steps: inquiring all target data and the main key thereof, taking out each piece of data, calculating CRC value, inquiring source data according to the main key, modifying the data according to configuration, comparing the data, storing the data which are failed to be compared into a test report, and finally refining inconsistent data in the test report into fields.
Through the steps, the data migration of the heterogeneous database is completed.
Referring to fig. 2, a block diagram of a data migration system based on heterogeneous databases according to an embodiment of the present invention is shown, where the system includes the following modules:
The environment configuration module is used for processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment;
The data migration module is used for importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file;
and the data verification module is used for verifying the migrated data.
The embodiment of the invention solves the problem of incompatibility of the formats of the data migration in the process of carrying out localized development on the software, ensures the consistency and the integrity of the data, avoids the problem of data loss or damage, and can carry out special configuration on specific tables and field modification according to the requirements so as to realize the customization processing of specific fields.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (10)
1. A heterogeneous database-based data migration method, comprising the steps of:
processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment;
importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file;
And verifying the migrated data.
2. The method for migrating data based on heterogeneous databases according to claim 1, wherein the method for processing the data in the database to be migrated and generating database objects, and modifying the table structure in the new database according to service variation or specification to obtain the database development environment comprises the following specific steps:
specifically, migrating the database table, and recording the relevant authorities of objects such as views, functions, triggers, synonyms and the like; implementing the method and the trigger by codes, using a Restful interface for external views, modifying a table structure according to service variation or specification problems, exporting the table and the views to generate SQL, combing, and generating database objects in a new database according to the SQL; the database is developed, the method and the trigger are realized by codes, the external view is realized by using a Restful interface, and the structure of the table is modified according to service variation or specification problems in the development process, so that the database development environment is obtained.
3. The data migration method based on heterogeneous databases according to claim 1, wherein the data is configured according to a migration tool, and the specific method comprises the following steps:
an automatic corresponding stage, wherein when the table names of the source database and the target database are the same, the data is automatically written into the corresponding table;
the default level of the system, the management system unifies the added fields or unifies the fields needing to be refreshed, and refreshes all the tables containing the fields;
The database object level is specially configured for specific table modification and specific field modification.
4. A heterogeneous database-based data migration method according to claim 3, wherein the data is automatically written into the corresponding table when the table names and field names of the source database and the target database are the same, and the specific method comprises the following steps:
inquiring table data from a source database, and when the data volume is too large, creating an intermediate table or carrying out paging inquiry and putting the intermediate table into a memory; inserting the data queried in the previous step into a target database according to the homonym table and the homonym; if there is paging, it is necessary to cyclically insert all the data in the table.
5. A heterogeneous database-based data migration method according to claim 3, wherein the management system unifies added fields or unifies fields to be refreshed, and refreshes all tables containing the fields, and the specific method comprises the following steps:
And (3) traversing all fields in the lookup table to query system-level configuration, wherein a table field containing the configuration selects a refreshing field according to a configuration rule or fills the redundant field with a value according to a value of a certain column in the table and table data of a source database.
6. A heterogeneous database-based data migration method according to claim 3, wherein the specific configuration is performed for specific table modification and specific field modification, and the specific method comprises the following steps:
The configuration is refreshed to the database, defaulting or modifying existing values for specific fields of a specific table.
7. The data migration method based on heterogeneous databases according to claim 1, wherein the verifying the migrated data comprises the following specific steps:
inquiring all target data and the main key thereof, taking out each piece of data, calculating CRC value, inquiring source data according to the main key, modifying the data according to configuration, comparing the data, storing the data which are failed to be compared into a test report, and finally refining inconsistent data in the test report into fields.
8. The heterogeneous database-based data migration method according to claim 5, wherein the selecting the refresh field according to the configuration rule comprises the following specific steps:
Refreshing a field so that the value of the field in each table is a certain value; if there is this field in the table, the value of this field needs to be modified all the way to the value of this field, and then the "if all updated" parameter is configured.
9. The method for migrating data based on heterogeneous databases according to claim 1, wherein the filling the redundant fields with values according to the values of a certain column in the table and the table data of the source database comprises the following specific steps:
If the table contains a certain field and other fields containing the field, the field values of the other fields containing the field need to be queried from the base table according to the field values, and the update target table is filled in.
10. A heterogeneous database-based data migration system, comprising the following modules:
The environment configuration module is used for processing the data in the database to be migrated, generating a database object, and modifying the table structure in the new database according to service variation or specification to obtain a database development environment;
The data migration module is used for importing the migration database object from the acquired development environment, and migrating the data according to the migration tool configuration and the corresponding migration configuration file;
and the data verification module is used for verifying the migrated data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410308033.6A CN117971809A (en) | 2024-03-18 | 2024-03-18 | Data migration method and system based on heterogeneous database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410308033.6A CN117971809A (en) | 2024-03-18 | 2024-03-18 | Data migration method and system based on heterogeneous database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117971809A true CN117971809A (en) | 2024-05-03 |
Family
ID=90854812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410308033.6A Pending CN117971809A (en) | 2024-03-18 | 2024-03-18 | Data migration method and system based on heterogeneous database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117971809A (en) |
-
2024
- 2024-03-18 CN CN202410308033.6A patent/CN117971809A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309196A (en) | Block chain data storage and query method, apparatus, equipment and storage medium | |
| CN103995854A (en) | Equipment cross-version upgrading method and device | |
| CN111078702B (en) | SQL sentence classification management and unified query method and device | |
| CN115328853A (en) | AUTOSAR-based file configuration method and device | |
| CN107016016A (en) | A kind of method and device of data processing | |
| US10776313B2 (en) | Converting source objects to target objects | |
| CN113177090A (en) | Data processing method and device | |
| CN107357588B (en) | Object code generation method and device | |
| CN105868127A (en) | Data storage method and device and data reading method and device | |
| US7325003B2 (en) | Method and system for mapping datasources in a metadata model | |
| CN117971809A (en) | Data migration method and system based on heterogeneous database | |
| CN108614869A (en) | A kind of method of historical data in different automation system platform synchronous migrations | |
| CN105389340A (en) | Information testing method and device | |
| CN115114325B (en) | Data query method and device, electronic equipment and storage medium | |
| CN106855821A (en) | A kind of distributed transaction processing method and device | |
| CN109636256B (en) | MAC address management method of MES system | |
| CN115168205A (en) | Automatic testing method and device | |
| CN114925142A (en) | Multi-type database compatibility method, device, equipment and medium of ORM framework | |
| CN107844639B (en) | Project standard structure automatic generation method and system | |
| CN111090638B (en) | Comparison method and device for transaction functions in database migration | |
| CN113703777A (en) | Code generation method and device based on database table, storage medium and equipment | |
| CN113177020A (en) | Data processing method and device | |
| CN113254455A (en) | Dynamic configuration method and device of database, computer equipment and storage medium | |
| CN111723104A (en) | Method, device and system for syntax analysis in data processing system | |
| CN116627390B (en) | ICD file substitution method and device in aviation software development |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |