CN114896196A - Data migration method and system for domestic database - Google Patents
Data migration method and system for domestic database Download PDFInfo
- Publication number
- CN114896196A CN114896196A CN202210399419.3A CN202210399419A CN114896196A CN 114896196 A CN114896196 A CN 114896196A CN 202210399419 A CN202210399419 A CN 202210399419A CN 114896196 A CN114896196 A CN 114896196A
- Authority
- CN
- China
- Prior art keywords
- database
- migration
- data
- domestic
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/119—Details of migration of file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application belongs to the field of database processing, and particularly relates to a domestic database-oriented data migration method and system. The method comprises the following steps: step S1, determining users, tables, data and view objects and corresponding permissions in a source database to be migrated; step S2, generating a corresponding database export file by using an export tool of the source database; step S3, converting the database export file into a text format SQL file; step S4, processing the text format SQL file into a structured file of the target domestic database in batch according to the data type mapping relation between the source database and the target domestic database; step S5, importing the structured file into the target domestic database which is preset with the same table space as the source database and users; and step S6, determining whether migration failure exists according to the migration log. The method and the device for migrating the data on demand have the advantages of high performance, high expandability and high availability.
Description
Technical Field
The application belongs to the field of database processing, and particularly relates to a data migration method and system for a domestic database.
Background
In recent years, with the continuous progress of information-oriented construction, the types and the number of application systems increase year by year, and the synchronous burst data makes management, daily maintenance and application have great challenges. In an actual production environment, the rapid increase of the data volume causes many problems of a relational database adopted in historical construction, such as slow data access speed, easy collapse of application and the like, the continuous enrichment of data generates huge demands on capacity, and the continuous stable operation demands promote the system to have higher availability; with the mass data brought by the high-speed development of 5G, IOT, the traditional centralized data storage has certain limitation, a database is required to support one-stop analysis and intelligent application, the characteristics of high availability of cloud, easiness in management and the like can be enjoyed, and the existing interface is compatible; in the national strategy, in order to meet the localization requirements of the standard country and simultaneously guarantee the informatization safety, reliability and autonomous controllability under a model design system, a business system related to the model needs to be switched into a domestic database and run in a domestic operating system, and by installing an autonomous middleware, the stable running (complete function and good performance) of the business system is ensured, and the data management service required by the model is provided.
At present, the migration mode of the domestic database mostly adopts a migration tool provided in a matching way to carry out automatic migration. Only source databases with high compatibility with the tool can be migrated by using the tool; the mapping relation of the database object is usually packaged in a tool and cannot be read, and when the migration fails, independent migration of views, packages, storage processes and functions which are possibly abnormal is difficult to carry out; and most of the scenes do not support heterogeneous synchronization and real-time synchronization.
Disclosure of Invention
In order to solve the problems, the application provides a data migration method and a data migration system facing to a domestic database, and a data migration process is realized by adopting a button and an Informatica.
The first aspect of the application provides a data migration method facing a domestic database, which mainly comprises the following steps:
step S1, determining users, tables, data and view objects and corresponding permissions in a source database to be migrated;
step S2, generating a corresponding database export file by using an export tool of the source database;
step S3, converting the database export file into a text format SQL file;
step S4, processing the text format SQL file into a structured file of the target domestic database in batch according to the data type mapping relation between the source database and the target domestic database;
step S5, importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and step S6, determining whether migration failure exists according to the migration log, and if the default migration sequence is not right, performing secondary migration on the objects with the dependency relationship.
Preferably, before step S1, the method further includes backing up data of the source database.
Preferably, after step S5, the method further includes:
and for the data which is not successfully exported or processed in a compatible way, carrying out secondary migration through an Informatica module of the ETL tool, carrying out supplementary processing on the constraint, the view and the index in the source database, and importing the data into the target database.
Preferably, the source database includes, but is not limited to mysql, oracle, sqlserver, and the target domestic database includes a domestic ZNBASE database.
A second aspect of the present application provides a data migration system for a domestic database, which mainly includes:
the source database object attribute acquisition module is used for determining the user, the table, the data and the view object in the source database to be migrated and the corresponding authority;
the source database file export module is used for generating a corresponding database export file by using an export tool of a source database;
the format conversion module is used for converting the database export file into a text format SQL file;
the batch processing module is used for batch processing the text format SQL file into a structured file of the target domestic database according to the data type mapping relation between the source database and the target domestic database;
the import module is used for importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and the secondary migration module is used for determining whether the migration failure condition exists according to the migration log, and for the objects with the dependency relationship, performing secondary migration if the default migration sequence is not correct.
Preferably, the system further comprises a backup module, configured to perform data backup on the source database.
Preferably, the system further comprises a supplementary migration module, which is used for performing secondary migration on data which is not successfully exported or subjected to compatibility processing through an information module of the ETL tool, performing supplementary processing on constraints, views and indexes in the source database, and importing the data into the target database.
Preferably, the source database includes, but is not limited to mysql, oracle, sqlserver, and the target domestic database includes a domestic ZNBASE database.
According to the method and the system, the data are migrated according to needs by accessing and integrating the relevant service system database and the data in different forms, and the method and the system have the characteristics of high performance, high expandability and high availability.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of the data migration method for domestic databases according to the present application.
Detailed Description
In order to make the implementation objects, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are implementations that are part of this application and not all implementations. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application, and should not be construed as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without any inventive work based on the embodiments in the present application are within the scope of protection of the present application. Embodiments of the present application will be described in detail below with reference to the drawings.
The first aspect of the application provides a data migration method facing a domestic database, which mainly comprises the following steps:
step S1, determining users, tables, data and view objects and corresponding permissions in a source database to be migrated;
step S2, generating a corresponding database export file by using an export tool of the source database;
step S3, converting the database export file into a text format SQL file;
step S4, processing the text format SQL file into a structured file of the target domestic database in batch according to the data type mapping relation between the source database and the target domestic database;
step S5, importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and step S6, determining whether migration failure exists according to the migration log, and if the default migration sequence is not right, performing secondary migration on the objects with the dependency relationship.
In some alternative embodiments, the source database includes, but is not limited to mysql, oracle, sqlserver, and the target domestic database includes a domestic ZNBASE database.
The data migration method and the data migration system are based on the mapping relation between the local relational database and the domestic ZBASE database data object, and realize the data migration facing the domestic ZBASE database in a mode of combining an ETL tool and a Python script. The domestic ZBASE database has the characteristics of strong consistency, high available distributed architecture, distributed horizontal expansion, high performance, enterprise-level safety and the like. Meanwhile, supporting services such as automatic operation and maintenance, monitoring and warning and the like are provided, and a complete distributed database solution is provided for users. ETL is an abbreviation for Extract-Transform-Load in english, and is used to describe the process of extracting (Extract), transposing (Transform), and loading (Load) data from a source end to a destination end.
The method mainly adopts the key and the information to realize the data migration process, and migrates data according to needs by accessing and integrating the related service system database and data in different forms, thereby having the characteristics of high performance, high expandability and high availability.
In some alternative embodiments, prior to step S1, it is further required to clarify prerequisites of data migration, such as: the method comprises the steps of source database type, source and target database server network communication condition, database objects and dependency relationship of users, tables, data, views and the like in a database to be migrated, authority condition of users for migration, source and target database object mapping relationship, source database backup and the like. After the prerequisites of data migration are determined, the specific corresponding database table space and users are established in the domestic database and authorized.
In step S4, an automatic processing program script is compiled based on the clear mapping relationship between the source database and the target database object, and the database file that is exported first is processed into a structured file that can be imported into the target database.
Finally, checking a migration log, judging whether an object which fails to be migrated exists, and if the default migration sequence of the object which has a dependency relationship is not right, performing secondary migration; and if the log has no errors, confirming that the number of the target database objects is the same as that of the source database, so as to avoid missing the migration objects in the migration process.
In some optional embodiments, after step S5, the method further includes:
and for the data which is not successfully exported or processed in a compatible mode, carrying out secondary migration through an information module of the ETL tool, carrying out supplementary processing on the constraint, the view and the index in the source database, and importing the data into the target database.
It should be noted that the migration method is implemented based on an import/export tool provided in the database, and may also be implemented by processing through an ETL tool, and implementing automatic replacement processing by embedding a mapping relationship into an ETL workflow process. Incremental synchronization can be realized through an ETL tool and the combination of unique attributes of database objects, and data can be migrated without interruption through the combination of data migration of full volume and incremental volume. Actually, for a database object with a large occupied space, a filtering condition can be added to split the database object, and the breakpoint continuous transmission capability is realized by combining with an ETL operation monitoring mechanism.
The application provides a specific embodiment for carrying out data migration from an Oracle database to a domestic ZBASE database, in the example, a source database selects to use the Oracle database 11g, a target database selects to use the domestic ZBASE database, and the detailed data migration step comprises the following steps:
1) confirming a firewall and a port strategy of a migration verification environment based on security and confidentiality requirements of a confidential network, and carrying out corresponding data backup in advance;
2) defining a migration task, and confirming database objects such as users, tables, data, views and the like in an Oracle database to be migrated and corresponding permissions;
3) exporting Oracle Schema by using an Oracle export tool;
4) converting the exported Schema file into a text format SQL file;
5) formatting an SQL (structured query language) file in an export text format as input, changing the corresponding replaceable contents in the input into corresponding ZNBASE database objects as output by combining the difference and data type mapping relation of Oracle and a domestic ZNBASE database, and writing the process into a file exported in the Python script batch processing step 4;
6) compressing the formatted file processed in the step 5 to improve the migration efficiency;
7) importing the formatted data compressed in the step 6 into a target database;
8) if the data cannot be successfully exported or processed in a compatible mode, secondary migration is attempted through an ETL tool Informatica;
9) performing supplementary processing on constraints, views, indexes and the like in a source database, and importing the constraints, the views, the indexes and the like into a target database;
10) comparing and checking the migration log, judging whether an object which fails to migrate exists, and if the default migration sequence is not right for the object which has a dependency relationship, performing secondary migration; and if no error exists in the log, confirming that the number of the target database objects is the same as that of the source database objects so as to avoid missing selection of the migration objects in the migration process.
A second aspect of the present application provides a data migration system for a domestic database, which mainly includes:
the source database object attribute acquisition module is used for determining the user, the table, the data and the view object in the source database to be migrated and the corresponding authority;
the source database file export module is used for generating a corresponding database export file by using an export tool of a source database;
the format conversion module is used for converting the database export file into a text format SQL file;
the batch processing module is used for batch processing the text format SQL file into a structured file of the target domestic database according to the data type mapping relation between the source database and the target domestic database;
the import module is used for importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and the secondary migration module is used for determining whether the migration failure condition exists according to the migration log, and for the objects with the dependency relationship, performing secondary migration if the default migration sequence is not correct.
In some optional embodiments, the system further includes a backup module, configured to perform data backup on the source database.
In some optional embodiments, the system further includes a supplementary migration module, configured to perform secondary migration on data that has not been successfully exported or subjected to compatibility processing through an information module of the ETL tool, perform supplementary processing on constraints, views, and indexes in the source database, and import the data into the target database.
In some alternative embodiments, the source database includes, but is not limited to mysql, oracle, sqlserver, and the target domestic database includes a domestic ZNBASE database.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (8)
1. A data migration method facing a domestic database is characterized by comprising the following steps:
step S1, determining users, tables, data and view objects and corresponding permissions in a source database to be migrated;
step S2, generating a corresponding database export file by using an export tool of the source database;
step S3, converting the database export file into a text format SQL file;
step S4, processing the text format SQL file into a structured file of the target domestic database in batch according to the data type mapping relation between the source database and the target domestic database;
step S5, importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and step S6, determining whether migration failure exists according to the migration log, and if the default migration sequence is not right, performing secondary migration on the objects with the dependency relationship.
2. The domestic database oriented data migration method of claim 1, further comprising, before step S1, performing data backup on said source database.
3. The domestic database oriented data migration method of claim 1, wherein after step S5, further comprising:
and for the data which is not successfully exported or processed in a compatible mode, carrying out secondary migration through an information module of the ETL tool, carrying out supplementary processing on the constraint, the view and the index in the source database, and importing the data into the target database.
4. The method for migrating data towards domestic databases of claim 1 wherein said source databases include, but are not limited to mysql, oracle, sqlserver and said target domestic database includes a domestic ZNBASE database.
5. A data migration system for domestic databases is characterized by comprising:
the source database object attribute acquisition module is used for determining the user, the table, the data and the view object in the source database to be migrated and the corresponding authority;
the source database file export module is used for generating a corresponding database export file by using an export tool of a source database;
the format conversion module is used for converting the database export file into a text format SQL file;
the batch processing module is used for batch processing the text format SQL file into a structured file of the target domestic database according to the data type mapping relation between the source database and the target domestic database;
the import module is used for importing the structured file into the target domestic database which is preset with the same table space as the source database and users;
and the secondary migration module is used for determining whether the migration failure condition exists according to the migration log, and for the objects with the dependency relationship, performing secondary migration if the default migration sequence is not correct.
6. A domestic database oriented data migration system according to claim 5, further comprising a backup module for performing data backup on said source database.
7. A domestic database-oriented data migration system as claimed in claim 5, further comprising a supplementary migration module, for performing secondary migration on data which has failed export or compatibility processing through an Informatica module of the ETL tool, performing supplementary processing on constraints, views and indexes in the source database, and importing the data into the target database.
8. The national database-oriented data migration system as claimed in claim 5, wherein said source database includes but is not limited to mysql, oracle, sqlserver, and said target national database includes a national ZBASE database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210399419.3A CN114896196A (en) | 2022-04-15 | 2022-04-15 | Data migration method and system for domestic database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210399419.3A CN114896196A (en) | 2022-04-15 | 2022-04-15 | Data migration method and system for domestic database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114896196A true CN114896196A (en) | 2022-08-12 |
Family
ID=82717594
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210399419.3A Pending CN114896196A (en) | 2022-04-15 | 2022-04-15 | Data migration method and system for domestic database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114896196A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303371A (en) * | 2023-05-17 | 2023-06-23 | 北京比格大数据有限公司 | Cross-architecture data computing method and device, electronic equipment and storage medium |
-
2022
- 2022-04-15 CN CN202210399419.3A patent/CN114896196A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116303371A (en) * | 2023-05-17 | 2023-06-23 | 北京比格大数据有限公司 | Cross-architecture data computing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108052681B (en) | Method and system for synchronizing structured data between relational databases | |
| CN107301250B (en) | Multi-source database collaborative backup method | |
| US9047392B2 (en) | System and method for conversion of JMS message data into database transactions for application to multiple heterogeneous databases | |
| US8832020B2 (en) | Method and system for safely transporting legacy data to an object semantic form data grid | |
| CN107391758B (en) | Database switching method, device and equipment | |
| CN111367886A (en) | Method and device for data migration in database | |
| CN111324610A (en) | Data synchronization method and device | |
| KR20060045622A (en) | Extraction, transformation and loading designer module of a computerized financial system | |
| CN104462269A (en) | Isomerous database data exchange method and system | |
| CN108229799B (en) | Multi-source heterogeneous power grid operation real-time data access system and method | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN114896196A (en) | Data migration method and system for domestic database | |
| CN107491558B (en) | Metadata updating method and device | |
| CN112699118A (en) | Data synchronization method, corresponding device, system and storage medium | |
| CN116414801A (en) | Data migration method, device, computer equipment and storage medium | |
| CN115114258A (en) | Data copying method and device, electronic equipment and computer storage medium | |
| CN112835887A (en) | Database management method, database management device, computing equipment and storage medium | |
| Kvet et al. | Data loading and migration methods in the cloud environment | |
| CN116450609B (en) | Method and equipment for modeling on heterogeneous data source based on unified grammar | |
| CN202084034U (en) | Distribution type searching system | |
| Liu et al. | A research on unified storage management and access technology applied in power network dispatch and control big data | |
| CN116860879A (en) | Intelligent substation control layer data high-speed real-time synchronization method and system | |
| Zhao et al. | A Nosql-Based Data Sharing Capability Mechanism in Smart Grid | |
| Schlarb et al. | Archival Information Package (AIP) Pilot Specification | |
| EA036384B1 (en) | Method and system for integrated management of big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |