CN112148709A - Data migration method, system and storage medium - Google Patents
Data migration method, system and storage medium Download PDFInfo
- Publication number
- CN112148709A CN112148709A CN202010952928.5A CN202010952928A CN112148709A CN 112148709 A CN112148709 A CN 112148709A CN 202010952928 A CN202010952928 A CN 202010952928A CN 112148709 A CN112148709 A CN 112148709A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- file
- type
- data migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 122
- 230000005012 migration Effects 0.000 title claims abstract description 122
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000002955 isolation Methods 0.000 abstract description 5
- 238000012545 processing Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 24
- 230000008676 import Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 9
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2107—File encryption
Abstract
The application discloses a data migration method, a data migration system and a storage medium, and relates to the technical field of data processing. The embodiment of the application comprises the following steps: acquiring a first database type of a source end database and a second database type of a target end database; calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file; and sending the data export file to a target end database. The method and the device can support intelligent data migration of isomorphic or heterogeneous databases, can automatically finish data export and data loading, have the characteristics of instant storage, data migration of the heterogeneous databases, cross-platform characteristics and safety isolation, and save the time and labor cost of data migration.
Description
Technical Field
The embodiment of the present application relates to, but is not limited to, the technical field of data processing, and in particular, to a data migration method, system and storage medium.
Background
Data is a core asset of enterprise operation and business development, and in the life cycle of enterprise application service and data transfer, there are often data migration requirements, such as building up test and development environments, upgrading software and hardware of a host, upgrading database software, and the like.
Disclosure of Invention
The present application aims to solve at least one of the technical problems in the prior art to some extent. Therefore, the embodiment of the application provides a data migration method, a data migration system and a storage medium, which can adaptively complete data migration according to the type of a database, and save the time and labor cost of data migration.
In a first aspect, an embodiment of the present application provides a data migration method, where the method is applied to a source database, and the method includes: acquiring a first database type of a source end database and a second database type of a target end database; calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file; and sending the data export file to a target end database.
The data migration method of the embodiment of the application has at least the following beneficial effects:
1. intelligence and self-adaptation. According to the data export requirement, the type of the database is intelligently identified, the applicable database data backup functional interface and the applicable database data backup functional component are matched and automatically called, and the data export process is simple, quick, automatic, intelligent, flexible and practical.
2. Heterogeneous and cross-platform support. For the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data hot standby and export can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In some embodiments, sending the data export file to the target database includes: encrypting the data export file to obtain an encrypted data file; and sending the encrypted data file to a target end database.
The data migration method of the embodiment of the application automatically performs encryption operation, avoids sensitive data leakage, and meets the safety requirement in data file transmission.
In a second aspect, an embodiment of the present application provides a data migration method, where the method is applied to a target-side database, and the method includes: acquiring a first database type of a source end database and a second database type of a target end database; acquiring a data export file from a source database; and calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
The data migration method of the embodiment of the application has at least the following beneficial effects:
1. intelligence and self-adaptation. According to the data import requirement, the type of the database is intelligently identified, the applicable database data loading functional interface and the applicable database data loading functional component are matched and automatically called, and the data import process is simple, quick, automatic, intelligent, flexible and practical.
2. Heterogeneous and cross-platform support. For the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In some embodiments, obtaining a data export file from a source database comprises: acquiring an encrypted data file from a source database; the encrypted data file is decrypted to obtain a data export file.
In some embodiments, the data migration method of the first or second aspect of the present application is run in a container.
In a third aspect, an embodiment of the present application provides a data migration system, where the system is applied to a source database, and the system includes: the database scanning module is used for acquiring a first database type of a source end database and a second database type of a target end database; and the data export module is connected with the database scanning module and used for calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file and sending the data export file to the target end database.
The data migration system of the embodiment of the application has at least the following beneficial effects:
1. the type of the database is intelligently identified through the database scanning module, and an applicable database data backup function interface and component are matched and automatically called through the data export module, so that the data export process is simple, quick, automatic, intelligent, flexible and practical;
2. for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), data hot standby and export are automatically completed by adopting a general data file format through the data export module, and the time and labor cost of data migration are saved.
In some embodiments, the data migration system further comprises: and the data encryption module is connected with the data export module and used for encrypting the data export file to obtain an encrypted data file and sending the encrypted data file to the target end database.
According to the data migration system, encryption operation is carried out through the data encryption module, sensitive data can be prevented from being leaked, and the security requirement in data file transmission is met.
In a fourth aspect, an embodiment of the present application provides a data migration system, where the system is applied to a target-side database, and the system includes: the database scanning module is used for acquiring a first database type of a source end database and a second database type of a target end database; and the data loading module is connected with the database scanning module and used for acquiring a data export file from the source database and calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
The data migration system of the embodiment of the application has at least the following beneficial effects:
1. the data loading module is matched with an applicable database data loading functional interface and component and automatically called, so that the data importing process is simple, rapid, automatic, intelligent, flexible and practical;
2. for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In some embodiments, the data migration system further comprises: and the data decryption module is connected with the data loading module and is used for acquiring the encrypted data file from the source database and decrypting the encrypted data file to acquire the data export file.
In a fifth aspect, embodiments of the present application provide a storage medium for a computer-readable storage, where the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the data migration method of the first aspect or the second aspect of the present application.
By executing the data migration method of the first aspect or the second aspect of the present application, the storage medium of the embodiment of the present application can intelligently identify the type of the database, match the type to an applicable database data backup or loading functional interface and component, and automatically call the type according to the requirement of data export or import, and the process of data export or import is simple, fast, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data hot standby, export or loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
FIG. 1 is a flow diagram of one embodiment of a method for data migration in an embodiment of the present application;
FIG. 2 is a flow diagram of another embodiment of a method for data migration in an embodiment of the present application;
FIG. 3 is a flow diagram of another embodiment of a method for data migration in an embodiment of the present application;
FIG. 4 is a flow chart of another embodiment of a method for data migration in an embodiment of the present application;
FIG. 5 is a flow chart of another embodiment of a method for data migration in an embodiment of the present application;
FIG. 6 is a block diagram of a particular embodiment of a data migration system in an embodiment of the present application.
Detailed Description
The conception and the resulting technical effects of the present application will be clearly and completely described below in conjunction with the embodiments to fully understand the objects, features and effects of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all embodiments, and other embodiments obtained by those skilled in the art without inventive efforts based on the embodiments of the present application belong to the protection scope of the present application.
It should be noted that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different from that in the flowcharts. The terms first, second and the like in the description and in the claims, and the drawings described above, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The methods disclosed in embodiments of the present application include one or more steps or actions for achieving the methods. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.
As used in the embodiments of the present application, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, studying, querying (e.g., querying in a table, a database, or another data structure), judging, and the like. Further, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and so forth. Further, "determining" may include resolving, selecting, choosing, establishing, and the like. The term "module" may, for example, mean a unit comprising one of, or a combination of two or more of, hardware, software, and firmware. A "module" may be used interchangeably with the terms "unit," "logic block," "component," or "circuit," for example. A "module" may be the smallest unit of integrated component elements or parts thereof. A "module" may be a minimal unit for performing one or more functions or portions thereof. The "module" may be implemented mechanically or electronically. For example, a "module" according to the present Application may include at least one of an Application-Specific Integrated Circuit (ASIC) chip, a Field-Programmable Gate Array (FPGA), and a Programmable logic device for performing operations that are known or will be developed later.
Structured Query Language (SQL) is a special-purpose programming Language, a database Query and programming Language, used to access data and Query, update, and manage relational database systems. Cold backup, also referred to as offline backup, refers to a full backup of a database performed in a situation where the database is closed and cannot be updated, and specified recovery of data may be performed. Logical hot backup refers to using a database tool to export data from a database and write the data into an output file, wherein the format of the file is generally different from that of the original database, only one image of the data content in the original database is suitable for export of data at a table level and a user level, and the file is not suitable for backup of a whole database.
Data is a core asset of enterprise operation and business development, and in the life cycle of enterprise application service and data transfer, there are often data migration requirements, such as building up test and development environments, upgrading software and hardware of a host, upgrading database software, and the like.
Data migration of a database needs to support both persistent and non-persistent data (i.e., runtime data stored in a container layer), so an online hot standby method is generally adopted to export data. The traditional manual migration method has the following difficulties and defects:
(1) the efficiency is low. The traditional database hot standby method is characterized in that environment variables are manually set and different hot standby commands are started or corresponding hot standby scripts are customized based on different types of database products and installation environments, the preparation time is long, the operation is complex, and the efficiency is low.
(2) The workload is large. For a target-end database running on a container platform, data loading needs to be completed by confirming relevant environment variables and calling corresponding data import interfaces, and finally import data needs to be verified, so that the whole process is complicated and the workload is large.
(3) There is a risk of data compromise. In the database data migration process based on the container platform, before data is exported from a source library, landed to a data file and loaded to a target library, a potential safety hazard exists, namely a risk of divulging a secret of a physical file containing the data exists.
(4) The migration process between heterogeneous databases or heterogeneous platforms is cumbersome. One difficulty of data migration is migration of data between heterogeneous databases or heterogeneous platforms, for example, a source database and a target database are different databases such as MySQL, Oracle, PostgreSQL, and the like, or are deployed on different platforms such as Linux and Windows, and conventional manual migration needs to prepare corresponding exporting and loading scripts to adapt to differences between the heterogeneous databases and the heterogeneous platforms, so that the preparation time is long, and the operation process is complicated.
Based on this, the embodiments of the present application provide a data migration method, system and storage medium, which can adaptively complete data migration according to a database type, and save time and labor cost for data migration.
In a first aspect, an embodiment of the present application provides a data migration method, where the method is applied to a source database, and with reference to fig. 1, the method includes the following steps:
s110, acquiring a first database type of a source end database and a second database type of a target end database;
s120, calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file;
and S130, sending the data export file to a target end database.
Step S110, automatically identifying the type of the database by adopting an intelligent scanning method, scanning the specific processes and ports of the database running at the source end and the target end, intelligently identifying the type of the database running at the source end and the target end, automatically collecting relevant environment variables, database software installation paths, relevant users and modes or table names needing to be migrated, and storing the information to prepare for data migration. And step S120, detecting whether the data migration is the same type of database data migration. If the databases are the same, the corresponding database special data backup interfaces are automatically called to complete data hot backup, and data export files are generated. For example, an Oracle database calls the exp program, a MySQL database calls the MySQL program, a PostgreSql database calls the pg _ dump program, a maridb calls the MySQL dump or the maridb-dump program. If the data migration between the heterogeneous databases is performed, a universal data backup interface (based on SQL language) specially developed for the data migration is automatically called to complete data hot backup, and a data export file is generated. In some embodiments, it is automatically determined whether the source-side database and the target-side database are both databases supported by the system, and if the databases at either end do not support, the program is exited. And if the databases at both ends support, detecting whether the databases are the same type of database data migration. Step S130, the source database sends the generated data export file to the target database, and completes the data migration of the source. The data migration method can intelligently scan and confirm the type of the database, automatically collect the environment variables, automatically call the database backup interface to complete data hot backup and export, is simple and easy to use, can be adapted to main-stream relational databases of MySQL, PostgreSQL, Oracle, MariaDB and the like, greatly reduces time and labor cost, and realizes intelligence and pipelining of IT operation and maintenance.
In some embodiments, referring to fig. 2, step S130 includes the steps of:
s210, encrypting the data export file to obtain an encrypted data file;
and S220, sending the encrypted data file to a target end database.
Step S210, the data export file is automatically scanned and identified, the data export file is automatically encrypted, the content of the data export file cannot be read under an unauthorized state, and the data security can be protected to the maximum extent. Step S220, the encrypted data export file is transmitted to the target end through the network, and if the environments of the source end and the target end are completely physically isolated, the data export file needs to be copied and transmitted to the target end.
In a second aspect, an embodiment of the present application provides a data migration method, where the method is applied to a target-side database, and referring to fig. 3, the method includes the following steps:
s310, acquiring a first database type of a source end database and a second database type of a target end database;
s320, acquiring a data export file from a source database;
s330, calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
Step S310, automatically identifying the type of the database by adopting an intelligent scanning method, scanning the specific processes and ports of the database operation of the source end and the target end, intelligently identifying the type of the database operation of the source end and the target end, automatically collecting relevant environment variables, database software installation paths, relevant users and modes or table names needing to be migrated, and storing the information to prepare for data migration. In step S320, the source database sends the generated data export file to the target database, and the target database receives the data export file from the source database. Step S330, whether the database data migration is the same type is detected according to the stored related information. If the databases are the same, the corresponding special data loading interfaces of the databases are automatically called to complete data loading. For example, an Oracle database calls the imp program, a MySQL database calls the MySQL program, a PostgreSql database calls the psql command line import, and maridb calls the MySQL command line import. And if the data migration between the heterogeneous databases is performed, automatically calling a general-purpose (SQL language-based) data loading interface to complete data loading. In some embodiments, the data load contains multi-scenario exception handling rules to address exceptions that may be encountered during the data load process. For example, when the data table to be imported exists in the target end database, the data table existing at the target end is renamed and backed up, and then the data table from the source end is imported; and when the insufficient storage space of the target end database is detected, calling the SQL statement to expand the related storage space and then importing the data file from the source end. In other embodiments, after the data loading is completed, the data import verification function is automatically called to ensure the integrity of the data. According to the data migration method, the type of the database can be intelligently identified, and the applicable database data loading function interface and component can be matched and automatically called according to the data import requirement, so that the data import process is simple, rapid, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In some embodiments, referring to fig. 4, step S320 includes the steps of:
s410, acquiring an encrypted data file from a source database;
s420, decrypting the encrypted data file to obtain a data export file.
Step S410, the encrypted data export file is transmitted from the source end to the target end, and the target end receives the encrypted data file. Step S420, the target decrypts the encrypted data file, and reads the encrypted data file in the authorized state to obtain a data export file. The data migration method can intelligently scan and identify the data to derive the file, authorize the decryption of the state file and protect the data security to the maximum extent.
In some embodiments, the data migration method of embodiments of the present application operates in a container.
Containers are lightweight operating system level virtualization technologies, one container containing a complete application runtime environment: all dependencies, class libraries, other binaries, configuration files, etc. required by the application, except the application itself, are uniformly typed into a package called a container image. When the mirror image is executed, the mirror image runs in an isolation environment and does not share the memory, the CPU and the disk of the host machine, so that the condition that the process in the container cannot monitor any process outside the container is ensured.
The use of the container technology has the following advantages:
(1) the performance is better. The operation container is equivalent to starting a process on the host machine operating system, so that the virtual machine is higher in starting speed, less in occupied resource and better in performance compared with the virtual machine.
(2) Is more agile. The container can run on all mainstream Linux distribution, Microsoft platforms and the like based on an open standard, and encapsulates all relevant details such as application dependence and an operating system which are necessary for running application programs, so that the image is more flexibly transplanted from one environment to another environment.
(3) And is safer. A container is an isolated environment, all required services can be provided by the container in a self-dependent mode, the isolation improves the safety property, meanwhile, the high-cohesion performance can realize rapid separation of problematic services, and rapid debugging and timely treatment can be realized in some complex systems.
The container was originally designed for stateless applications: a service is created with the container and destroyed after use. When using a container to deploy a database, configuration data persistence needs to be considered, i.e., data can still be retained after the container is destroyed. Docker is a current mainstream open source container application engine, and many mainstream relational and non-relational databases can be conveniently and rapidly operated on a Docker container platform. In Docker, the most common data persistence method is Docker Volume, which manages a directory of a host and then maps and mounts the directory into a container. The Docker mirror image is formed by overlapping a plurality of file systems (read-only layers), and when a container is started, the Docker loads the read-only mirror image layer and adds a read-write layer on the read-only mirror image layer. When the Docker container is deleted and restarted through the mirror, the previous changes will be lost. The combination of the read-only layer and the read-write layer on top is called a Union File System. To be able to hold data as well as data between shared containers, Docker proposes the concept of Volume. Volume is a directory or file that may exist on the host in the form of a normal file or directory, bypassing the default federated file system. The Docker Volume can be used to save and share data, both in the domain and lifecycle independently of a single container.
In some embodiments, referring to fig. 5, a data migration method includes: and starting intelligent database scanning, and detecting whether the source end database and the target end database are both supported database types. And if at least one of the source end database and the target end database is of an unsupported database type, ending the data migration process. And if the source end database and the target end database are both supported database types, starting data derivation and detecting whether the source end database and the target end database are isomorphic databases. If the source end and the target end databases are isomorphic databases, calling a database special data backup interface to finish data migration; and if the source end database and the target end database are heterogeneous databases, calling a universal data backup interface of the databases to finish data migration. And after the database interface is called, generating a data export file. Encrypting the data export file, starting data loading, decrypting the encrypted data export file, and detecting whether the source end database and the target end database are isomorphic databases. If the source end and the target end databases are isomorphic databases, calling a special data loading interface of the databases to finish data migration; and if the source end database and the target end database are heterogeneous databases, calling a universal data loading interface of the databases to finish data migration. And after the data loading is finished, verifying the loaded data to ensure the integrity of the data.
The data migration method can realize online migration of data, support intelligent data migration of isomorphic or heterogeneous databases, automatically complete data export and data loading, has the characteristics of instant storage, data migration of the heterogeneous databases, cross-platform characteristics and safety isolation, and saves time and labor cost for data migration.
In a third aspect, an embodiment of the present application provides a data migration system, where the system is applied to a source database, and referring to fig. 6, the system 600 includes: the database scanning module 610, the database scanning module 610 is used for obtaining a first database type of the source-end database and a second database type of the target-end database; and a data export module 620 connected to the database scanning module 610, wherein the data export module 620 is configured to call a dedicated data backup interface or a general data backup interface according to the first database type and the second database type to perform data hot standby and generate a data export file, and send the data export file to the target database.
The database scanning module automatically identifies the type of the database by adopting an intelligent scanning method, scans the specific processes and ports of the database running at the source end and the target end, intelligently identifies the types of the database running at the source end and the target end, automatically collects relevant environment variables, database software installation paths, relevant users and modes or table names needing to be migrated, and stores the information to prepare for data migration. The data export module detects whether the data migration is the same type of database data migration. If the databases are the same, the corresponding database special data backup interfaces are automatically called to complete data hot backup, and data export files are generated. For example, an Oracle database calls the exp program, a MySQL database calls the MySQL program, a PostgreSql database calls the pg _ dump program, a maridb calls the MySQL dump or the maridb-dump program. If the data migration between the heterogeneous databases is performed, a universal data backup interface (based on SQL language) specially developed for the data migration is automatically called to complete data hot backup, and a data export file is generated. In some embodiments, the data export module automatically determines whether the source database and the target database are both databases supported by the system, and exits the program if the databases at either end do not support. And if the databases at both ends support, detecting whether the databases are the same type of database data migration. And the source end database sends the generated data export file to the target end database to complete the data migration of the source end. The data migration system of the embodiment of the application intelligently identifies the type of the database through the database scanning module, and is matched with an applicable database data backup functional interface and component through the data export module and automatically called, so that the data export process is simple, rapid, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (Windows/Linux), data hot standby and export are automatically completed by adopting a general data file format through the data export module, and the time and labor cost of data migration are saved.
In some embodiments, the data migration system further comprises: and the data encryption module is connected with the data export module and used for encrypting the data export file to obtain an encrypted data file and sending the encrypted data file to the target end database. The data encryption module automatically performs encryption operation, can avoid sensitive data leakage and meets the safety requirement in data file transmission.
In a fourth aspect, an embodiment of the present application provides a data migration system, where the system is applied to a target-side database, and the system includes: the database scanning module is used for acquiring a first database type of a source end database and a second database type of a target end database; and the data loading module is connected with the database scanning module and used for acquiring a data export file from the source database and calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
The database scanning module automatically identifies the type of the database by adopting an intelligent scanning method, scans the specific processes and ports of the database running at the source end and the target end, intelligently identifies the types of the database running at the source end and the target end, automatically collects relevant environment variables, database software installation paths, relevant users and modes or table names needing to be migrated, and stores the information to prepare for data migration. The source end database sends the generated data export file to the target end database, and the target end database receives the data export file from the source end database. And the data loading module detects whether the data migration is the same type of database data migration or not according to the stored related information. If the databases are the same, the corresponding special data loading interfaces of the databases are automatically called to complete data loading. For example, an Oracle database calls the imp program, a MySQL database calls the MySQL program, a PostgreSql database calls the psql command line import, and maridb calls the MySQL command line import. And if the data migration between the heterogeneous databases is performed, automatically calling a general-purpose (SQL language-based) data loading interface to complete data loading. In some embodiments, the data loading module completing the data loading further comprises setting an exception handling rule of multiple scenarios to solve an exception condition that may be encountered during the data loading process. For example, when the data table to be imported exists in the target end database, the data table existing at the target end is renamed and backed up, and then the data table from the source end is imported; and when the insufficient storage space of the target end database is detected, calling the SQL statement to expand the related storage space and then importing the data file from the source end. In other embodiments, after the data loading is completed, the data loading module automatically calls a data import verification function to ensure the integrity of the data. The data migration system of the embodiment of the application is matched with an applicable database data loading functional interface and an applicable database data loading functional component through the data loading module and automatically called, and the data importing process is simple, quick, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In some embodiments, the data migration system further comprises: and the data decryption module is connected with the data loading module and is used for acquiring the encrypted data file from the source database and decrypting the encrypted data file to acquire the data export file.
The encrypted data export file is transmitted from the source end to the target end, and the target end receives the encrypted data file. And the data decryption module decrypts the encrypted data file, reads the encrypted data file in an authorized state and obtains a data export file. The data migration system of the embodiment of the application authorizes the state file to decrypt through the data decryption module, and protects the data security to the maximum extent.
The data migration system of the embodiment of the application can realize online migration of data, support intelligent data migration of isomorphic or heterogeneous databases, automatically complete data export and data loading, has the characteristics of instant storage, data migration of the heterogeneous databases, cross-platform characteristics and safety isolation, and saves time and labor cost of data migration.
In a fifth aspect, embodiments of the present application provide a storage medium for a computer-readable storage, where the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to implement the data migration method of the first aspect or the second aspect of the present application.
Storage media for embodiments of the present application may include computer storage media (or non-transitory media) and communication media (or transitory media). Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
By executing the data migration method of the first aspect or the second aspect of the present application, the storage medium of the embodiment of the present application can intelligently identify the type of the database, match the type to an applicable database data backup or loading functional interface and component, and automatically call the type according to the requirement of data export or import, and the process of data export or import is simple, fast, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data hot standby, export or loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
In a sixth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and when the program is executed by the processor, the data migration method of the first aspect or the second aspect of the present application is implemented.
In the electronic device of the embodiment of the present Application, the Processor may be a Central Processing Unit (CPU), or may be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center of the operable device for a data migration method, and various interfaces and lines connecting the various parts of the operable device for the entire data migration method.
The memory may be used to store computer programs and/or modules, and the processor may implement the various functions of the executable device of a data migration method by executing or executing the computer programs and/or modules stored in the memory, as well as invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The electronic device may include, for example, at least one of a smart phone, a tablet, a Personal Computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook, a Computer, a workstation, a server, a Personal Digital Assistant (PDA), a Portable Multimedia Player (PMP), an MPEG-1 audio layer 3(MP3) Player, a mobile medical device, a camera, and a wearable device. The wearable Device may include at least one of an accessory type (e.g., watch, ring, bracelet, foot chain, necklace, glasses, contact lens, or Head-Mounted Device (HMD)), a fabric or garment integration type (e.g., electronic garment), a body-Mounted type (e.g., skin pad or tattoo), and a bio-implantable type (e.g., implantable circuitry).
By executing the data migration method of the first aspect or the second aspect of the present application, the electronic device of the embodiment of the present application can intelligently identify the type of the database, match the type to an applicable database data backup or loading functional interface and component, and automatically call the type according to the requirement of data export or import, and the process of data export or import is simple, fast, automatic, intelligent, flexible and practical; for the situation that the source end database and the target end database do not belong to the same database or are deployed on different platforms (such as Windows/Linux), the data hot standby, export or loading can be automatically completed by adopting a general data file format, and the time and labor cost of data migration are saved.
The embodiments of the present application have been described in detail with reference to the drawings, but the present application is not limited to the embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present application. Furthermore, the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
Claims (10)
1. A data migration method, wherein the method is applied to a source database, and wherein the method comprises:
acquiring a first database type of the source end database and a second database type of the target end database;
calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file;
and sending the data export file to the target end database.
2. The data migration method according to claim 1, wherein the sending the data export file to the target database includes:
encrypting the data export file to obtain an encrypted data file;
and sending the encrypted data file to the target end database.
3. The data migration method is characterized in that the method is applied to a target-end database, and comprises the following steps:
acquiring a first database type of a source end database and a second database type of a target end database;
acquiring a data export file from the source database;
and calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
4. The data migration method of claim 3, wherein the obtaining the data export file from the source database comprises:
acquiring an encrypted data file from the source database;
and decrypting the encrypted data file to obtain the data export file.
5. The data migration method according to any one of claims 1 to 4, wherein the method is run in a container.
6. A data migration system, wherein the system is applied to a source database, the system comprising:
a database scanning module, configured to obtain a first database type of the source database and a second database type of the target database;
and the data export module is connected with the database scanning module and used for calling a special data backup interface or a general data backup interface according to the first database type and the second database type to complete data hot standby and generate a data export file, and sending the data export file to the target end database.
7. The data migration system of claim 6, wherein the system further comprises:
and the data encryption module is connected with the data export module and used for encrypting the data export file to obtain an encrypted data file and sending the encrypted data file to the target end database.
8. The data migration system is characterized in that the system is applied to a target-end database, and the system comprises:
a database scanning module, configured to obtain a first database type of a source database and a second database type of the target database;
and the data loading module is connected with the database scanning module and used for acquiring a data export file from the source database and calling a special data loading interface or a general data loading interface according to the first database type and the second database type to read the data export file and complete data loading.
9. The data migration system of claim 8, wherein the system further comprises:
and the data decryption module is connected with the data loading module and is used for acquiring the encrypted data file from the source database and decrypting the encrypted data file to acquire the data export file.
10. A storage medium for computer readable storage, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the data migration method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010952928.5A CN112148709A (en) | 2020-09-11 | 2020-09-11 | Data migration method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010952928.5A CN112148709A (en) | 2020-09-11 | 2020-09-11 | Data migration method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112148709A true CN112148709A (en) | 2020-12-29 |
Family
ID=73890204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010952928.5A Pending CN112148709A (en) | 2020-09-11 | 2020-09-11 | Data migration method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112148709A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312147A (en) * | 2021-06-11 | 2021-08-27 | 中国科学技术大学 | Method and system for migrating object storage across cluster mass data |
| CN113965340A (en) * | 2021-08-30 | 2022-01-21 | 广东南方通信建设有限公司 | Cross-platform data migration method, system and readable medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106682534A (en) * | 2017-01-23 | 2017-05-17 | 郑州云海信息技术有限公司 | Method and device for verifying data integrity in data migration process |
| US20180246886A1 (en) * | 2017-02-27 | 2018-08-30 | OSF Global Services Inc. | Data migration for platform integration |
| US10185727B1 (en) * | 2015-12-30 | 2019-01-22 | Amazon Technologies, Inc. | Migrating data between databases |
| CN110147411A (en) * | 2019-05-20 | 2019-08-20 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111104393A (en) * | 2019-12-17 | 2020-05-05 | 深圳追一科技有限公司 | Data migration method and device, electronic equipment and storage medium |
-
2020
- 2020-09-11 CN CN202010952928.5A patent/CN112148709A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10185727B1 (en) * | 2015-12-30 | 2019-01-22 | Amazon Technologies, Inc. | Migrating data between databases |
| CN106682534A (en) * | 2017-01-23 | 2017-05-17 | 郑州云海信息技术有限公司 | Method and device for verifying data integrity in data migration process |
| US20180246886A1 (en) * | 2017-02-27 | 2018-08-30 | OSF Global Services Inc. | Data migration for platform integration |
| CN110147411A (en) * | 2019-05-20 | 2019-08-20 | 平安科技（深圳）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
| CN111104393A (en) * | 2019-12-17 | 2020-05-05 | 深圳追一科技有限公司 | Data migration method and device, electronic equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312147A (en) * | 2021-06-11 | 2021-08-27 | 中国科学技术大学 | Method and system for migrating object storage across cluster mass data |
| CN113312147B (en) * | 2021-06-11 | 2022-12-30 | 中国科学技术大学 | Method and system for migrating object storage across cluster mass data |
| CN113965340A (en) * | 2021-08-30 | 2022-01-21 | 广东南方通信建设有限公司 | Cross-platform data migration method, system and readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10616194B2 (en) | Secure data destruction in a distributed environment using key protection mechanisms | |
| US20190187980A1 (en) | Version control of applications | |
| JP6166839B2 (en) | System and method for replacing application methods at runtime | |
| US9747438B2 (en) | Enabling resource access for secure application containers | |
| US20150081644A1 (en) | Method and system for backing up and restoring a virtual file system | |
| US9733927B2 (en) | Detection of software or hardware incompatibilities in software packages | |
| US20130036431A1 (en) | Constraining Execution of Specified Device Drivers | |
| US11593091B2 (en) | Method and apparatus for upgrading firmware of transfer device on mobile carrier, and non-transitory storage medium | |
| CN109753347B (en) | System and method for realizing driving | |
| US20170115980A1 (en) | Computerized system and method for patching an application by separating executables and working data using different images | |
| CN112148709A (en) | Data migration method, system and storage medium | |
| CN114925084B (en) | Distributed transaction processing method, system, equipment and readable storage medium | |
| CN109614203B (en) | Android application cloud data evidence obtaining and analyzing system and method based on application data simulation | |
| TW201351194A (en) | Data protection method for portable electronic device and computer program product for the same | |
| CN115629971A (en) | Application development system and method | |
| CN110704392A (en) | Block chain network upgrading method and device, storage medium and electronic equipment | |
| CN110990807B (en) | Method and equipment for encrypting and decrypting mobile terminal | |
| US10255435B1 (en) | Systems and methods for establishing a reputation for related program files | |
| CN109871327B (en) | Method and device for testing security of secure storage of trusted execution environment | |
| US20140258720A1 (en) | Systems and methods for transparent per-file encryption and decryption via metadata identification | |
| CN105574425B (en) | Access the method and device of storage data | |
| US11288361B1 (en) | Systems and methods for restoring applications | |
| CN112650713A (en) | File system operation method, device, equipment and storage medium | |
| CN112231757B (en) | Privacy protection method, device and equipment for embedded application | |
| CN116700768B (en) | Application processing method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201229 |