WO2023134627A1

WO2023134627A1 - Data migration methods and apparatuses, electronic device, and storage medium

Info

Publication number: WO2023134627A1
Application number: PCT/CN2023/071319
Authority: WO
Inventors: 王增臣; 沈文宽; 侯博文; 付裕; 黄健
Original assignee: 中兴通讯股份有限公司
Priority date: 2022-01-13
Filing date: 2023-01-09
Publication date: 2023-07-20
Also published as: CN114077602B; CN114077602A

Abstract

Embodiments of the present application relate to the technical field of databases and provide data migration methods and apparatuses, an electronic device, and a storage medium. The data migration method comprises: acquiring source data from an original database (S210); receiving migration request information sent by a client end, the migration request information comprising a data attribute parameter of the source data (S220); generating a migration policy according to the data attribute parameter (S230); and migrating the source data to a destination database according to the migration policy (S240).

Description

Data migration method and device, electronic device, storage medium

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202210035298.4 and a filing date of January 13, 2022, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The embodiments of the present application relate to the technical field of databases, and in particular, to a data migration method and device, electronic equipment, and storage media.

Background technique

The development of information services has brought about an increasing amount of data, and databases play an indispensable role as a data bridge in information systems. For example, a distributed database is a logically unified database composed of multiple physically dispersed database units connected by a computer network. It has the characteristics of large storage capacity, high concurrent business volume, and good scalability, so its application is increasingly widely. In the application scenarios of distributed databases, cross-database migration of data is a common operation, which puts forward higher requirements for reliable data migration technology.

The current data migration method mainly adopts simple multi-threaded migration. The database of the original database is migrated by multi-threaded data according to the table or file strategy. The uneven data of different source tables or files leads to a long overall migration time and low migration efficiency. lower.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

Embodiments of the present application provide a data migration method and device, electronic equipment, and a storage medium.

In the first aspect, the embodiment of the present application provides a data migration method, which is applied to the migration terminal. The method includes: obtaining source data from the original database; receiving migration request information sent by the client; the migration request information includes the A data attribute parameter of the source data; a migration strategy is generated according to the data attribute parameter; and the source data is migrated to a target database according to the migration strategy.

In the second aspect, the embodiment of the present application provides a data migration method, which is applied to a client, and the method includes: obtaining migration request information; the migration request information includes the data attribute parameters of the source data; sending the migration request information to the migrating end, so that the migrating end executes the method described in the first aspect of the present application.

In the third aspect, the embodiment of the present application provides a data migration device for the migration terminal, including: a source data acquisition module, configured to acquire source data from the original database; a migration request information receiving module, configured to receive the client The migration request information sent; the migration request information includes the data attribute parameters of the source data; the migration strategy generation module is configured to generate a migration strategy according to the data attribute parameters; the migration module is configured to generate a migration strategy according to the migration strategy Migrate the source data to the target database.

In a fourth aspect, an embodiment of the present application provides a data migration device for a client, including: a data acquisition module configured to acquire migration request information; the migration request information includes data attribute parameters of the source data; data The sending module is configured to send the migration request information to the migration end, so that the migration end executes the method described in the first aspect of the present application.

In the fifth aspect, the embodiment of the present application provides an electronic device, including: at least one memory; at least one processor; at least one program; the program is stored in the memory, and the processor executes the at least one program to realize: Apply the method as described in the first aspect above; or, the method as described in the second aspect above.

In a sixth aspect, the embodiment of the present application provides a storage medium, the storage medium is a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to make the computer perform: The method as described in the first aspect above; or, the method as described in the second aspect above. It can be understood that the beneficial effects of the above-mentioned second aspect to the sixth aspect compared with the related technology are the same as those of the above-mentioned first aspect compared with the related technology. Please refer to the relevant description in the above-mentioned first aspect. This will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the embodiments or related technical descriptions. Obviously, the drawings in the following description are only the embodiments of the present application For some embodiments of the present invention, those of ordinary skill in the art can also obtain other drawings based on these drawings without paying creative efforts.

FIG. 1 is a schematic diagram of a system architecture of a data migration method provided by an embodiment of the present application.

FIG. 2 is a flowchart of a data migration method provided by an embodiment of the present application.

Fig. 3 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 4 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 5 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 6 is a flowchart of a data migration method provided by another embodiment of the present application.

FIG. 7 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 8 is a flowchart of a data migration method provided by another embodiment of the present application.

FIG. 9 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 10 is a flowchart of a data migration method provided by another embodiment of the present application.

FIG. 11 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 12 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 13 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 14 is a flowchart of a data migration method provided by another embodiment of the present application.

Fig. 15 is a flowchart of a data migration method provided by another embodiment of the present application.

FIG. 16 is a schematic diagram of the principle of the data migration method provided by the embodiment of the present application.

FIG. 17 is a schematic diagram of a data migration module provided by an embodiment of the present application.

Fig. 18 is a schematic diagram of a data migration module provided by another embodiment of the present application.

Reference numerals: original database 100, client 110, migration terminal 120, strategy optimization module 121, data export module 122, data import module 123, result output module 124, destination database 130, source data acquisition module 1710, migration request information reception Module 1720 , migration policy generation module 1730 , migration module 1740 , data acquisition module 1810 , and data sending module 1820 .

Detailed ways

In the following description, for the purpose of illustration rather than limitation, details such as specific system structures and technologies are presented so as to thoroughly understand the embodiments of the present application. However, it should be apparent to those skilled in the art that the embodiments of the present application can also be practiced in other embodiments without these details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the embodiments of the present application with unnecessary detail.

It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than in the flowchart. The terms "first", "second" and the like in the specification and claims and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

It should also be understood that references to "one embodiment" or "some embodiments" described in the description of the embodiments of the present application mean that specific features described in conjunction with the embodiments of the present application are included in one or more embodiments of the embodiments of the present application. , structure or characteristics. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless specifically stated otherwise.

Database (Database, DB) is a warehouse that organizes, stores and manages data according to the data structure. It is a collection of large amounts of data stored in a computer for a long time, organized, shareable, and uniformly managed. Database Management System (DBMS) is a large-scale software for manipulating and managing databases, used to create, use and maintain databases. It manages and controls the database in a unified way to ensure the security and integrity of the database. Data migration (Data Migration, DM) generally refers to the process of converting and migrating the source data of the old database management system to the new database management system. The data migration tool is an important auxiliary tool to ensure the integrity and consistency of the data and effectively complete the replacement and upgrade of the database system.

Distributed database refers to the use of smaller computer systems, each computer can be placed in a separate place, each computer may have a complete copy of the DBMS, or a partial copy of the copy, and has its own local database, located in Many computers in different locations are connected to each other through the network to form a complete and global logically centralized and physically distributed large database.

The current data migration method mainly adopts simple multi-threaded migration. The database of the original database is migrated by multi-threaded data according to the table or file strategy. The migration efficiency of processing massive data is still not up to expectations. Different source tables or files The unevenness of the data leads to a long overall migration time and low migration efficiency.

In addition, the current data migration method does not consider the processing of cross-database field type conversion, resulting in a decrease in the accuracy of data migration; the current data migration method is mainly applicable to general database data migration scenarios, and cannot realize data migration to distributed databases , and did not consider the distribution and distribution of data on the distributed database. When the customer replaces and upgrades the database system, it is urgent to complete the migration of massive data from the Oracle database to the new distributed database in a short period of time. At the same time, selective migration of some data cannot be realized, so the flexibility of data migration is very poor.

Based on this, an embodiment of the present disclosure provides a data migration method and device, electronic equipment, and a storage medium. At the migration end, the source data from the original database is obtained, and the migration request information sent by the client is received, wherein the migration request information includes the source data According to the data attribute parameters, a migration strategy is generated according to the data attribute parameters, and then the source data is migrated to the destination database according to the migration strategy. On the client side, the migration request information is obtained and sent to the migration end to complete the migration task. This application is implemented The technical solution provided by the example can realize the multi-process and multi-thread migration of the database, avoid the problem of long migration time caused by uneven data, and improve the efficiency of data migration; realize the processing of cross-database field type conversion, improve It improves the accuracy of data migration; realizes data migration to distributed databases, considers the distribution and distribution of data on distributed databases, and realizes the selective migration of some data at the same time, which improves the flexibility of data migration.

Embodiments of the present disclosure provide a data migration method and device, electronic equipment, and a storage medium, which are described through the following embodiments. First, the data migration method in the embodiments of the present disclosure is described.

The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

As shown in FIG. 1 , FIG. 1 is a schematic diagram of a system architecture of a data migration method provided by an embodiment of the present application. In Fig. 1, the system architecture includes an original database 100, a client 110, a migration terminal 120, and a destination database 130, wherein the migration terminal 120 includes a strategy optimization module 121, a data export module 122, a data import module 123, and a result output module 124 .

Wherein, the original database 100 is connected with the data export module, the client 110 is connected with the strategy optimization module 121, the strategy optimization module 121 is connected with the data export module 122, the data export module 122 is connected with the data import module 123, and the data import module 123 is connected with the target database 130, and the result output module 124 is connected to the data export module 122 and the data import module 123 respectively.

In this embodiment, the original database 100 is a database to be migrated, and the client 110 is set to send migration request information to the strategy optimization module 121, so that the strategy optimization module 121 can generate a migration strategy according to the migration request information, and then according to The data in the original database 100 is exported by the migration strategy in the data export module 122 to obtain an intermediate file, and then the data is imported in the data import module 123 according to the migration strategy, and the intermediate file is distributed and imported into the destination database 130 to complete the migration process. In addition, the result output module 124 records the intermediate file in the data export module 122 and the migration process file in the data import module 123 to generate a migration result report file, so as to record the migration result.

In one embodiment, the target database 130 is a distributed database platform, and the migration terminal 120 is an import service platform, such as an Oraclecap platform. Among them, the distributed database platform is the entity and core of the existing distributed database, which is mainly responsible for the management and monitoring of data storage and system status; the data import service platform Oraclecap provides data export from the oral database to CSV files, and then through CSV The files are imported into the internal distributed database system.

The system architecture and application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the system architecture The evolution of the technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

Those skilled in the art can understand that the system architecture shown in Figure 1 does not constitute a limitation to the embodiment of the present application, and may include more or less components than those shown in the illustration, or combine some components, or different components layout.

In the system architecture shown in FIG. 1 , each device can call its stored data migration program to execute the data migration method.

Based on the above system architecture, various embodiments of the data migration method according to the embodiments of the present application are proposed.

The embodiment of the present application provides a data migration method, which is applied to the migration terminal. The method includes: obtaining the source data from the original database; receiving the migration request information sent by the client; the migration request information includes the data attribute parameters of the source data; according to the data attribute Parameters generate a migration strategy; migrate the source data to the destination database according to the migration strategy.

In one embodiment, the data attribute parameters include one or more of the following: table quantity parameters, or table data quantity parameters, or row quantity parameters.

As shown in Figure 2, Figure 2 is a flow chart of a data migration method provided by an embodiment of the present application, the data migration method is applied to the migration end, for example, it is applied to the migration end 120 in the embodiment shown in Figure 1, the The data migration method includes but not limited to step S210 and step S240.

S210, acquire source data from the original database;

S220. Receive migration request information sent by the client; the migration request information includes data attribute parameters of source data;

S230, generating a migration strategy according to the data attribute parameter;

S240. Migrate the source data to the target database according to the migration strategy.

In step S210, the source data is data in the original database, and the original database is directly connected to the migration end for sending the source data to the migration end for the migration end to migrate the source data.

In step S220, the migration request information is used by the client to transmit commands and parameters related to the migration process to the migration end, so that the migration platform performs migration processing on the source data according to the above commands and parameters. Wherein, the commands and parameters related to the migration process include but are not limited to data attribute parameters and authority authentication requests.

In step S230, the data attribute parameters include but are not limited to one or more of the following: table quantity parameters, or table data quantity parameters, or row quantity parameters. In an embodiment, the table quantity parameters represent the data table in the source data Quantity, the table data quantity parameter represents the size of the data table in the source data, and the row quantity parameter represents the setting value of each row of data. Data attribute parameters can be used both to generate migration strategies and to manipulate data during export and import.

In step S240, according to the migration strategy generated in step S230, the migration terminal is instructed to migrate the source data, so that the data can be imported into the target database reasonably and efficiently.

In the data migration method provided by this application, at the migration end, the source data from the original database is obtained, and the migration request information sent by the client is received, wherein the migration request information includes the data attribute parameters of the source data, and a migration strategy is generated according to the data attribute parameters, Furthermore, the source data is migrated to the destination database according to the migration strategy. Through the technical solution provided by the embodiment of the present application, the multi-process and multi-thread migration of the database can be realized, which avoids the problem of long migration time caused by uneven data, and improves the The efficiency of data migration; realize the processing of cross-database field type conversion, improve the accuracy of data migration; realize the data migration to distributed databases, consider the distribution and distribution of data on distributed databases, and realize the partial Selective migration of data improves the flexibility of data migration.

In addition, the data migration method provided in this application is only to export and import the source data, which will not cause damage to the original database environment, and also ensures the data validity of the target database.

In one embodiment, the migration strategy includes an export strategy and an import strategy; migrating the source data to the destination database according to the migration strategy includes: converting the source data into an intermediate file according to the export strategy; and migrating the intermediate file to the destination database according to the import strategy.

In addition, in an embodiment, referring to FIG. 3, step S240 may include but not limited to the following steps:

S310, converting the source data into an intermediate file according to the export policy;

S320. Migrate the intermediate file to the destination database according to the import policy.

In one embodiment, the migration terminal first exports the source data, homogenizes the data according to the export strategy, generates an intermediate file, and then distributes the data to the intermediate file according to the import strategy, and completes the migration of the data to the destination database. The data migration of distributed databases avoids the problem of long migration time caused by uneven data, improves the efficiency of data migration, and makes the data migration requirements between various databases have certain adaptability and reliability. scalability.

In one embodiment, the export strategy includes export-in thread strategy; converting the source data into an intermediate file according to the export strategy, including: configuring the number of export processes and the number of export threads according to the data attribute parameters; according to the number of export processes, the number of export threads and the data Attribute parameters, evenly distribute the data to the corresponding export threads for data export, and generate intermediate files.

In addition, in an embodiment, referring to FIG. 4, step S310 may include but not limited to the following steps:

S410, configure the number of export processes and the number of export threads according to the data attribute parameters;

S420. According to the number of export processes, the number of export threads, and the data attribute parameters, evenly distribute the data to corresponding export threads for data export, and generate an intermediate file.

In step S410, the data attribute parameters include but are not limited to table quantity parameters and table data volume parameters. In the embodiment, the table quantity parameter represents the number of data tables in the source data, and the table data volume parameter represents the size of the data tables in the source data , the number of rows parameter represents the setting value of each row of data.

In some embodiments, for data whose table number parameter is 700 and table data volume parameter is 10 billion, the number of export processes is configured to be 5, and the number of export threads is configured to be 20. Among them, the table number parameter is 700 means that the data center has 700 tables, and the table data volume parameter is 10 billion means that there are 10 billion pieces of data in the data.

In step S420, the data attribute parameter includes a row number parameter. In some embodiments, the row number parameter is 1 million, indicating that the fixed data row is set to 1 million rows of data during the data export process, and the fixed data row of the table data Split threads, realize concurrent export and generate intermediate files.

It should be noted that the setting of the table quantity parameter, the table data quantity parameter and the row quantity parameter can be changed according to different source data, as long as the corresponding technical effect can be achieved.

In one embodiment, the export strategy includes a data placement strategy, and converting the source data into an intermediate file according to the export strategy includes: obtaining a preset file division rule; the file division rule includes generating the data volume or data of each intermediate file The number of rows; according to the file division rules, the source data is correspondingly exported as at least one intermediate file.

In addition, in an embodiment, referring to FIG. 5, step S310 may include but not limited to the following steps:

S510, acquiring a preset file division rule;

S520. Correspondingly export the source data into at least one intermediate file according to the file division rule.

In step S510, the file division rule includes generating the data volume or data row number of each intermediate file, in some embodiments, generating the data volume number of each intermediate file is 64M, generating the data row number of each intermediate file It is 100,000 lines. It should be noted that the setting of the amount of data or the number of data lines for generating each intermediate file can be changed according to the source data, as long as the corresponding technical effect can be achieved.

In step S520, according to the file division rules, the source data is correspondingly exported into at least one intermediate file, the process of data storage is completed, and the uniform export of data is realized, which provides preparatory conditions for the rapid import of data, and can also facilitate The export process converts the format of the file. In addition, the generated intermediate file can be used not only for the data migration process, but also for the generation of the migration result report file, so as to realize the analysis and recording of the migration result.

In one embodiment, the import strategy includes import thread strategy; the intermediate file is migrated to the destination database according to the import strategy, including: configuring the number of import processes and the number of import threads according to the data attribute parameters; according to the number of import processes, the number of import threads and the data Attribute parameters, evenly distribute the data to the corresponding import threads for data import, and migrate the intermediate files to the destination database.

In addition, in one embodiment, referring to FIG. 6, step S320 may include but not limited to the following steps:

S610, configuring the number of import processes and the number of import threads according to the data attribute parameters;

S620. According to the number of import processes, the number of import threads and the data attribute parameters, evenly distribute the data to the corresponding import threads for data import, and migrate the intermediate files to the destination database.

In step S610, the data attribute parameters include but are not limited to table number parameters and table data volume parameters. In some embodiments, the table number parameters represent the number of data tables in the intermediate file, and the table data volume parameters represent the number of data tables in the source data. The size, number of rows parameter characterizes the setting value of each row of data.

In some embodiments, for data whose table number parameter is 700 and table data volume parameter is 10 billion, the number of import processes is configured to be 5 and the number of import threads to be 20. Among them, the table number parameter is 700 means that the data center has 700 tables, and the table data volume parameter is 10 billion means that there are 10 billion pieces of data in the data.

In step S420, the data attribute parameter includes a row quantity parameter. In some embodiments, the row quantity parameter is 5000, indicating that the file data is divided into threads every 5000 pieces, and the fixed data rows of the table data are split into threads to realize Concurrent imports to the destination database.

In one embodiment, generating the migration strategy according to the data attribute parameters includes: performing prediction processing according to the table quantity parameter and the table data quantity parameter to obtain intermediate parameters; and generating an import thread strategy according to the intermediate parameters.

In addition, in one embodiment, referring to FIG. 7, step S230 may include but not limited to the following steps:

S710, performing prediction processing according to the table quantity parameter and the table data quantity parameter, to obtain intermediate parameters;

S720. Generate an imported thread policy according to the intermediate parameter.

In step S710, the generation of the migration strategy is completed before the data export, including the export strategy and the import strategy, because the generation of the import strategy needs to depend on the number of data tables and the size of the data in the intermediate file, so it needs to be based on the source data before data import. The size of the data volume predicts the size of the intermediate file to obtain intermediate parameters for importing into the generation of the thread strategy, wherein the intermediate parameters include but are not limited to the table quantity parameters and table data volume parameters of the predicted intermediate files.

It should be noted that the prediction process in step S710 includes but is not limited to manual prediction and machine learning prediction. Among them, manual prediction is based on the staff's previous experience in data migration and the size of the generated intermediate file based on the size of the source file to predict the size of the generated intermediate file, obtain intermediate parameters, and input it into the system; machine prediction includes the use of machine learning, Deep learning and other methods train the prediction model through historical configuration data, and use the prediction model to realize the prediction process of intermediate parameters.

In step S720, after the intermediate parameters are predicted, the imported thread policy is configured using the intermediate parameters. The principle of the configuration is the same as the principle of configuring the exported thread policy according to the data attribute parameters.

In one embodiment, the destination database includes multiple partitions; the import strategy includes a data distribution strategy; migrating the intermediate file to the destination database according to the import strategy includes: obtaining distribution key information; according to the distribution key information, obtaining and matching the distribution key information The intermediate file; migrate the intermediate file to the corresponding partition of the destination database.

In an embodiment, the distribution key information includes at least any one of the following: a continuous interval distribution key, an enumeration value distribution key, or a partition distribution key.

In addition, in one embodiment, referring to FIG. 8, step S320 may include but not limited to the following steps:

S810, acquiring distribution key information;

S820. According to the distribution key information, obtain an intermediate file matching the distribution key information;

S830. Migrate the intermediate file to a partition corresponding to the destination database.

In step S810, the distribution key information is used to characterize the data distribution type for different types of data tables, and the distribution key information includes at least any of the following: continuous interval distribution key, or enumerated value distribution key, or partition distribution key, Among them, the continuous interval distribution key includes but is not limited to RANGE, which assigns data to different partitions based on a given continuous interval range; the enumerated value distribution key includes but is not limited to LIST, which partitions based on the enumerated value list; partition distribution Keys include but are not limited to HASH, which assigns data to different partitions based on a given number of partitions.

It should be noted that there are two ways to obtain distribution key information, namely direct acquisition and indirect acquisition. For basic data, the corresponding distribution key information can be obtained by directly matching the attributes of the basic data table; for business data In other words, you need to rely on the use of basic data for the selection of distribution keys.

In steps S820 to S830, the distribution key information has its own distribution rules. The core principle of the distribution key information is to migrate the matched intermediate files to the corresponding partitions of the destination database according to different distribution rules, thus realizing the distribution on the destination database. Distribution, which distributes business-related data to the same database unit, makes the data distribution reasonable and even after data migration, and improves the flexibility of data migration.

In one embodiment, the intermediate file includes basic data and business data, and obtaining the distribution key information also includes: obtaining the priority of the basic data; sorting the basic data according to the priority to obtain the sorting result; sorting the key according to the sorting result Segment query matching processing to obtain the basic data associated with the business data; obtain the distribution key corresponding to the basic data associated with the business data.

In addition, in an embodiment, referring to FIG. 9, step S810 may include but not limited to the following steps:

S910, acquiring the priority of the basic data;

S920, sorting the basic data according to the priority to obtain a sorting result;

S930. Perform key field query and matching processing according to the sorting results to obtain basic data associated with the business data;

S940. Obtain a distribution key corresponding to the basic data associated with the service data.

In some embodiments, the intermediate file includes basic data and business data, the basic data includes but not limited to the basic information table, and the business data includes but not limited to the business information table. In some embodiments, the basic information table refers to a table that stores data such as user basic information, and has the characteristics of being dependent and relatively stable data. Among them, some field information of the table can distinguish and reflect the characteristics of the table data, such as gender, place of origin and other information ; The business table is the business information data generated by the user in the process of handling business, which has a high correlation with the basic information table, and is generally related to the information of one or more basic information tables.

In step S910, the priority of the basic data is one of the attributes of the basic data, and is used for query matching according to the priority, so as to process the basic data with higher importance in advance and ensure the reliability of data distribution. For example, the priorities of the basic tables are: jc1, jc2, jc3.

From step S920 to step S940, query and match the key fields of the business data according to the priority order of the basic information table. The distribution key of the table data; if the basic information table cannot be matched, the distribution key of the business table data is set to the default value, for example, the default value can be hash(00001), where hash indicates that the distribution key is HASH.

In one embodiment, the migration request information also includes platform operating environment information, and converting the source data into an intermediate file according to the export strategy includes: generating a format conversion strategy according to the platform operating environment information; performing format conversion processing on the source data according to the format conversion strategy , get the target format data; update the intermediate file according to the target format data.

In addition, in one embodiment, referring to FIG. 10, step S310 may include but not limited to the following steps:

S1010, generating a format conversion strategy according to the platform operating environment information;

S1020, performing format conversion processing on the source data according to the format conversion strategy to obtain target format data;

S1030, updating the intermediate file according to the target format data.

In some embodiments, after the export process is completed, it is necessary to consider whether the format of the intermediate file meets the requirements of the current operating platform, so as to convert the format of the intermediate file. Generally, the process of format conversion occurs during the data export process.

In step S1010, the platform operating environment information is the operating environment information of the migration platform currently used by the migration terminal, and the format conversion strategy is the corresponding relationship between different platform operating environments and file formats. Different types of platforms have different requirements for file formats, so It is necessary to convert the file type of the source data according to the characteristics of the platform to improve the efficiency of data migration.

In steps S1020 to S1030, the target format data includes but is not limited to CSV files. It should be noted that the format of the target format data can also be adaptively adjusted according to the characteristics of other platforms or databases, as long as the corresponding technical effects can be achieved. For example, convert varchar2 type to varchar type; or convert number type to decimal type, etc.

In one embodiment, the migration request information further includes data screening information, and generating a migration strategy according to data attribute parameters includes: performing data source screening processing according to source data and data screening information to obtain target data; generating Migration strategy; correspondingly, migrating the source data to the destination database according to the migration strategy, including: migrating the target data to the destination database according to the migration strategy.

In addition, in one embodiment, referring to FIG. 11, step S230 may include but not limited to the following steps:

S1110. Perform data source screening processing according to the source data and data screening information to obtain target data;

S1120, generating a migration strategy according to the data attribute parameters of the target data;

Correspondingly, step S240 may include but not limited to the following steps:

S1130. Migrate the target data to the destination database according to the migration policy.

In step S1110, in some data migration scenarios, it is not necessary to migrate all the source data of the original database, but it is necessary to selectively migrate the required data or key data, wherein the target data is selected from the source data The required data or key data that come out.

In steps S1120 to S1130, the generation process of the migration strategy and the principle of data migration have not changed due to the change of the migration object. Customers can select the required or key data for migration according to business needs, so as to achieve the data cleaning effect on the original database and maintain This ensures high data availability in the destination database and improves the flexibility of the data migration process.

In one embodiment, the migration request information also includes a permission authentication request, and the migration policy is generated according to the data attribute parameters, including: parsing the permission authentication request to obtain permission information; performing authentication processing on the permission information to obtain authentication result information; Result information and data attribute parameters generate a migration strategy.

In addition, in one embodiment, referring to FIG. 12, step S230 may include but not limited to the following steps:

S1210, analyzing the permission authentication request to obtain permission information;

S1220. Perform authentication processing on the authority information to obtain authentication result information;

S1230. Generate a migration policy according to the authentication result information and data attribute parameters.

In step S1210, the user sends migration request information to the migration terminal through the client, wherein the migration terminal includes but is not limited to a data migration tool, and the data migration tool parses the data migration command to obtain permission information, and the permission information is used to indicate whether the client has Within the authority of data migration.

In step S1220, after obtaining the authority information, the migrating end performs authentication processing on the authority information to obtain authentication result information, which is used to represent the migrating end's authentication result of the client's authority.

In step S1230, if the authentication result is successful, that is, after the authority authentication is passed, the migration terminal generates a migration policy according to the authentication result information and data attribute parameters.

In an embodiment, the method further includes: generating a migration result report file according to the intermediate file.

In addition, in one embodiment, referring to FIG. 13 , the data migration method further includes:

S1310, acquiring intermediate files and migration process files;

S1320. Generate a migration result report file according to the intermediate file and the migration process file.

In step S1310, the intermediate file is an intermediate file obtained during the export process, the migration process file is during the export and import process, and the migration process file is a record of whether the export and import of each file is successful or not.

In step S1320, the intermediate files and migration process files are collected to generate a migration result report file for reuse in data analysis and the like. For example, the migration process file is used to judge whether each data table in the data source has been successfully migrated. If the migration is not successful, it can be judged by the recorded intermediate file, which step in the migration process of the data table has an error, and then for Data tables that have not been successfully migrated are migrated twice to ensure the reliability of data migration.

In one embodiment, the target database is a distributed database.

In some embodiments, the target database is a distributed database, which is used when the database system is replaced and upgraded, and the data in the original database is migrated to the distributed database system as needed. At this time, no matter whether the original database is a distributed database , or common databases, the data migration method of this application can be used to complete the migration process.

It should be noted that if both the original database and the target database are common databases, the data distribution function of the data migration method provided by this application is omitted, and the task of data migration can still be completed, which improves the universality of the data migration method of this application. fitness.

The embodiment of the present application provides a data migration method, which is applied to the client. The method includes: obtaining migration request information; the migration request information includes the data attribute parameters of the source data; sending the migration request information to the migration end, so that the migration end performs the above tasks A method for migrating endpoints.

As shown in FIG. 14, FIG. 14 is a flowchart of a data migration method provided by an embodiment of the present application. The data migration method is applied to a client, for example, to the client 110 in the embodiment shown in FIG. 1. The data migration method includes but not limited to step S1410 and step S1420.

S1410, acquiring migration request information;

S1420. Send migration request information to the migration end.

In step S1410, the migration request information is used by the client to transmit commands and parameters related to the migration process to the migration end, so that the migration platform performs migration processing on the source data according to the above commands and parameters. Wherein, the commands and parameters related to the migration process include but are not limited to data attribute parameters and authority authentication requests.

In step S1420, the migration request information is sent to the migration end, so that the migration end executes any of the above-mentioned methods for the migration end.

In view of the technical problems existing in the above-mentioned related technologies, in this embodiment, the migration request information is obtained at the client end and sent to the migration end to complete the migration task. The technical solution provided by the embodiment of this application can realize the database Multi-process and multi-thread migration avoids the problem of long migration time caused by uneven data and improves the efficiency of data migration; realizes the processing of cross-database field type conversion and improves the accuracy of data migration; The data migration of distributed databases considers the distribution and distribution of data on distributed databases, and at the same time realizes the selective migration of some data, which improves the flexibility of data migration.

Referring to FIG. 15 , the data migration method is described in detail in an embodiment. It should be understood that the following description is only an illustration, rather than a specific limitation on the application.

S1510, the client acquires and sends migration request information to the migration terminal;

S1520. The migration terminal acquires source data from the original database, and receives migration request information sent by the client; the migration request information includes data attribute parameters and data screening information of the source data;

S1530. The migration terminal parses the permission authentication request to obtain permission information, and then performs authentication processing on the permission information to obtain authentication result information;

S1540, if the migration terminal determines that the authentication result information is legal; then perform data source screening processing according to the source data and data screening information to obtain the target data;

S1550. The strategy optimization module generates an export strategy and an import strategy according to the table quantity parameter and the table data quantity parameter of the target data; wherein, the export strategy includes the number of export processes and the number of export threads.

S1560. The data export module configures the number of export processes and the number of export threads according to the data attribute parameters, evenly distributes the data to corresponding export threads for data export, and correspondingly exports the source data into at least one intermediate file according to file division rules.

S1570. The data export module generates a format conversion strategy according to the platform operating environment information, performs format conversion processing on the source data according to the format conversion strategy, obtains data in a target format, and then uses the data in the target format to update the intermediate file.

S1580, the data import module configures the number of import processes and the number of import threads according to the data attribute parameters, evenly distributes the data to corresponding import threads for data import, uses the distribution key information to obtain an intermediate file matching the distribution key information, and migrates the intermediate file to The partition corresponding to the destination database.

S1590, the result output module acquires the intermediate file and the migration process file, and generates a migration result report file.

In some embodiments, the target data table number parameter is 700, the table data volume parameter is 10 billion, the configuration export process number and import process number are 5, and the export thread number and import thread number are 20.

During the export process, the thread distribution is performed with a fixed data line of 1 million lines, and the data is placed to disk with a fixed line number of 100,000 lines to generate a csv intermediate file.

During the import process, 5000 data records are used for thread distribution, and the basic data and business data are deployed in the distributed database according to the corresponding distribution rules according to the distribution key information such as RANGE, LIST, and HASH.

In practical applications, the data migration process shown in Figure 15 occupies a total of 4T table space in the Oracle database, and completes all data migration within 3 hours, greatly improving the efficiency of data migration.

The following application example illustrates the data migration method of this application.

A typical data migration process is shown in Figure 16, and a brief description is as follows:

The original database (the source database in the figure) sends massive data to the data export module, and the data export module sends a policy query command to the policy optimization module. After receiving the query command, the policy optimization module returns policy information to the data export module, and then, the data The export module exports the source data into intermediate files according to the policy information, and the intermediate files are sequentially recorded as TB1.csv1...TBm.csv1, TBn.csv1...TBn.csvk, and then the data import module distributes the intermediate files to the destination database according to the policy information , the files of the target database are recorded as TB1 import...TBm import, TBn import...TBn import, and then the data migration is completed. Finally, the result output module forms a result file according to the data migration record, and the result file is recorded as result.

In addition, an embodiment of the embodiment of the present application also provides a data migration device for the migration terminal. Referring to FIG. 17, the data migration device includes:

source data acquisition module 1710, configured to acquire source data from the original database;

The migration request information receiving module 1720 is configured to receive the migration request information sent by the client; the migration request information includes the data attribute parameters of the source data;

The migration strategy generation module 1730 is configured to generate a migration strategy according to the data attribute parameters;

The migration module 1740 is configured to migrate the source data to the target database according to the migration strategy.

The source data acquisition module 1710 is connected to the migration request information receiving module 1720 , the migration request information receiving module 1720 is connected to the migration strategy generation module 1730 , and the migration strategy generation module 1730 is connected to the migration module 1740 .

It should be noted that the data migration device in this embodiment and the data migration method for the migration end belong to the same idea, so these embodiments have the same implementation principle and technical effect, and will not be described in detail here.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, an embodiment of the embodiment of the present application also provides a data migration device for a client. Referring to FIG. 18, the data migration device includes:

The data acquisition module 1810 is configured to acquire migration request information; the migration request information includes data attribute parameters of source data;

The data sending module 1820 is configured to send migration request information to the migration end, so that the migration end performs any one of the above-mentioned methods for the migration end in this application.

Wherein, the data acquisition module 1810 is connected with the data sending module 1820 .

It should be noted that the data migration device in this embodiment and the data migration method for the client belong to the same idea, so these embodiments have the same implementation principle and technical effect, and will not be described in detail here.

In addition, an embodiment of the embodiments of the present application further provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and operable on the processor.

The processor and memory can be connected by a bus or other means.

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may include memory located remotely from the processor, which remote memory may be connected to the processor via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

It should be noted that the electronic device in this embodiment can be applied as the client 110 or the migration terminal 120 in the system architecture of the embodiment shown in FIG. 1 . The electronic device in this embodiment is as shown in FIG. 2 The data migration methods in the embodiments have the same idea, so these embodiments have the same implementation principles and technical effects, and will not be described in detail here.

The non-transitory software programs and instructions required to implement the data migration method of the above-mentioned embodiments are stored in the memory, and when executed by the processor, the data migration method in any of the above-mentioned embodiments is executed.

In addition, an embodiment of the embodiment of the present application also provides a storage medium, the storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by the Execution by one processor may cause the above-mentioned processor to execute the data migration method in the above-mentioned embodiments, for example, execute the data migration method in any of the above-described embodiments.

Compared with related technologies, the data migration method provided by the first aspect of the embodiment of the present application uses the migration request information sent by the client, wherein the migration request information includes the data attribute parameters of the source data, and generates a migration strategy according to the data attribute parameters, and then Migrate the source data to the destination database according to the migration strategy. Through the technical solution provided by the embodiment of the present application, the multi-process and multi-thread migration of the database can be realized according to the data attribute parameters, avoiding the problem of long migration time caused by uneven data, and improving the efficiency of data migration.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is a description of some embodiments of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalents without violating the spirit of the embodiments of the present application. Modifications or replacements, these equivalent modifications or replacements are all included within the scope defined in the claims of the embodiments of the present application.

Claims

A data migration method comprising:

Get the source data from the original database;

receiving migration request information sent by the client; the migration request information includes data attribute parameters of the source data;

generating a migration strategy according to the data attribute parameters;

The source data is migrated to the target database according to the migration strategy.
The method according to claim 1, wherein the data attribute parameters include one or more of the following: table quantity parameters, or table data quantity parameters, or row quantity parameters.
The method according to claim 2, wherein the migration strategy includes an export strategy and an import strategy; and migrating the source data to the destination database according to the migration strategy includes:

converting the source data into an intermediate file according to the export strategy;

The intermediate file is migrated to the target database according to the import policy.
The method according to claim 3, wherein said export policy includes exporting into a thread policy; said converting said source data into an intermediate file according to said export policy comprises:

Configure the number of export processes and the number of export threads according to the data attribute parameters;

According to the number of export processes, the number of export threads, and the data attribute parameters, evenly distribute data to corresponding export threads for data export, and generate the intermediate file.
The method according to claim 3, wherein the export strategy includes a data placement strategy, and converting the source data into an intermediate file according to the export strategy includes:

Acquiring preset file division rules; the file division rules include generating the number of data volumes or the number of data lines for each of the intermediate files;

Correspondingly exporting the source data into at least one intermediate file according to the file division rule.
The method according to claim 3, wherein the import strategy includes importing a thread strategy; and migrating the intermediate file to the destination database according to the import strategy comprises:

Configure the number of import processes and the number of import threads according to the data attribute parameters;

According to the number of import processes, the number of import threads and the data attribute parameters, evenly distribute data to corresponding import threads for data import, and migrate the intermediate files to the target database.
The method according to claim 6, wherein said generating a migration strategy according to said data attribute parameters comprises:

performing prediction processing according to the table quantity parameter and the table data quantity parameter to obtain intermediate parameters;

Generate the imported thread policy according to the intermediate parameters.
The method according to claim 3, wherein the target database includes a plurality of partitions; the import policy includes a data distribution policy; and migrating the intermediate file to the target database according to the import policy includes:

Obtain distribution key information;

Acquiring the intermediate file matching the distribution key information according to the distribution key information;

Migrating the intermediate file to the partition corresponding to the target database.
The method according to claim 8, wherein the intermediate file includes basic data and business data, and the obtaining distribution key information further includes:

Acquiring the priority of the basic data;

sorting the basic data according to the priority to obtain a sorting result;

performing key field query matching processing according to the sorting results to obtain the basic data associated with the business data;

Acquiring the distribution key corresponding to the basic data associated with the service data.
The method according to any one of claims 8 or 9, wherein the distribution key information includes at least any one of the following: a continuous interval distribution key, an enumerated value distribution key, or a partition distribution key.
The method according to claim 3, wherein the migration request information further includes platform operating environment information, and converting the source data into an intermediate file according to the export strategy includes:

generating a format conversion policy according to the platform operating environment information;

performing format conversion processing on the source data according to the format conversion strategy to obtain target format data;

The intermediate file is updated according to the target format data.
The method according to claim 1, wherein the migration request information further includes data screening information, and generating a migration strategy according to the data attribute parameters includes:

performing data source screening processing according to the source data and the data screening information to obtain target data;

generating the migration strategy according to the data attribute parameters of the target data;

Correspondingly, the migrating the source data to the destination database according to the migration strategy includes:

Migrate the target data to the target database according to the migration strategy.
The method according to claim 1, wherein the migration request information further includes a permission authentication request, and generating a migration strategy according to the data attribute parameters includes:

Analyzing the permission authentication request to obtain permission information;

Perform authentication processing on the authority information to obtain authentication result information;

The migration policy is generated according to the authentication result information and the data attribute parameters.
The method according to claim 3, wherein the method further comprises:

A migration result report file is generated according to the intermediate file.
The method according to claim 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13 or 14, wherein the target database is a distributed database.
A data migration method comprising:

Obtain migration request information; the migration request information includes data attribute parameters of the source data;

Sending the migration request information to the migration end, so that the migration end executes the method according to any one of claims 1-15.
A data migration device, comprising:

a source data acquisition module configured to acquire source data from the original database;

The migration request information receiving module is configured to receive the migration request information sent by the client; the migration request information includes the data attribute parameters of the source data;

A migration strategy generation module, configured to generate a migration strategy according to the data attribute parameters;

The migration module is configured to migrate the source data to the target database according to the migration strategy.
A data migration device, comprising:

A data acquisition module configured to acquire migration request information; the migration request information includes data attribute parameters of the source data;

The data sending module is configured to send the migration request information to the migration end, so that the migration end executes the method according to any one of claims 1 to 15.
An electronic device comprising:

at least one memory;

at least one processor;

at least one program;

The programs are stored in memory, and the processor executes the at least one program to:

A method as claimed in any one of claims 1 to 15;

Or, the method of claim 16.
A storage medium, the storage medium is a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to make a computer perform:

A method as claimed in any one of claims 1 to 15;

Or, the method of claim 16.