CN113901031A

CN113901031A - Database migration method, electronic device and storage medium

Info

Publication number: CN113901031A
Application number: CN202111225730.8A
Authority: CN
Inventors: 董浩成; 张日康
Original assignee: Zhangyue Technology Co Ltd
Current assignee: Ireader Technology Co Ltd; Zhangyue Technology Co Ltd
Priority date: 2021-10-21
Filing date: 2021-10-21
Publication date: 2022-01-07

Abstract

The invention discloses a database migration method, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, based on a data migration setting interface; creating a conversion relation between a first data field in at least one source data table and a second data field in a target data table; and responding to conversion operation triggered by a user, and reading a first data field in the source data table according to the conversion relation so as to realize that the first data field is converted and migrated to a second data field of a target data table of the target database. The invention provides visual data migration through the data migration setting interface, can quickly finish the migration of the data in the source database to the target database by setting the conversion relation among the data, does not need to manually compile data conversion codes, is quicker and more convenient, and reduces the processing difficulty.

Description

Database migration method, electronic device and storage medium

Technical Field

The invention relates to the field of databases, in particular to a database migration method, electronic equipment and a storage medium.

Background

The database migration is to move data stored in one database to another database, when data migration is performed between different types of databases, for example, a source database is a non-relational database, and a target database is a relational database, because the types of the databases are different, data cannot be directly migrated to a table, and often a user needs to manually write corresponding data migration codes for a data table in the source database, and migrate the data to the target database by using the codes. Considering that the data volume in the database is large, the fields in different data tables are different, and the code corresponding to one data table cannot be multiplexed into other data tables, so that the code development amount of a user is large, and the data migration efficiency is low.

Disclosure of Invention

In view of the above, the present invention has been made to provide a database migration method, an electronic device, and a storage medium that overcome or at least partially solve the above-mentioned problems.

According to an aspect of the present invention, there is provided a database migration method, including:

acquiring at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, based on a data migration setting interface;

creating a conversion relation between a first data field in at least one source data table and a second data field in a target data table;

and responding to conversion operation triggered by a user, and reading a first data field in the source data table according to the conversion relation so as to realize that the first data field is converted and migrated to a second data field of a target data table of the target database.

According to another aspect of the present invention, there is provided an electronic apparatus including: the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the following operations:

According to another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to any one of the database migration methods described above.

According to the database migration method, the electronic device and the storage medium provided by the invention, visual data migration is provided through the data migration setting interface, the migration of data in the source database to the target database can be quickly completed by setting the conversion relation among the data, the data conversion code does not need to be manually written, the data migration method is quicker and more convenient, and the processing difficulty is reduced.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 illustrates a flow diagram of a database migration method according to one embodiment of the present invention;

FIG. 2 illustrates a data migration setup interface diagram;

FIG. 3 shows a flow diagram of a database migration method according to another embodiment of the invention;

FIG. 4 is a diagram illustrating a data storage structure of a column cluster database;

FIG. 5a shows a schematic flow diagram of a database migration method according to yet another embodiment of the invention;

FIG. 5b illustrates a data migration state information presentation diagram;

FIG. 6a shows a schematic flow diagram of a database migration method according to yet another embodiment of the invention;

FIG. 6b shows a data migration error information presentation diagram;

FIG. 7 illustrates a flow diagram of a database migration method according to yet another embodiment of the present invention;

fig. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the invention.

Detailed Description

Example one

Fig. 1 is a flow chart illustrating a database migration method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:

step S101, at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, are obtained based on a data migration setting interface.

The embodiment can be executed on a client of a user (database migration processing personnel), provides a visual data migration setting interface for the user, facilitates the user to directly operate and complete data migration in the data migration setting interface, does not need to realize the data migration through codes, reduces the technical threshold for the user, is also suitable for data migration of different databases and different data tables, has high multiplexing expansibility, and improves the data migration efficiency.

The data migration setting interface may be as shown in fig. 2, and the user may directly input the information of the source database to be migrated and the information of the specified target database. In this embodiment, the source database and the target database are databases of different types, the source database includes databases such as a column cluster database and an hbase, and the target database may be a non-column cluster database such as a relational database, mysql, mongdb, and the like, but not limited to the above types of databases, and different types of databases may be selected according to implementation conditions to implement data migration. To facilitate a better understanding of the present invention, the following description will be given by taking a cluster database as an example. In fig. 2, the user may specify the source data table of the source database to be migrated, for example, each data table in the source database is shown in the data migration interface, and the user may directly select one of the data tables as the source data table, or receive the hbase data table input by the user as the source data table, and the like. The target database is appointed by a user, and information such as a name space, an IP address, a library name, a url address, a login user name, a password, an appointed target data table and the like of the target database is input in a data migration interface, so that connection with the target database is established according to the information, and data migration is realized.

Step S102, creating a conversion relation between a first data field in at least one source data table and a second data field in a target data table.

The source data table comprises a plurality of first data fields, a conversion relation is established between each first data field to be migrated and each second data field in the target data table, and a data conversion basis between the first data field and the second data field is established, so that data migration is performed. The conversion relationship may be 1 to 1, 1 to many, many to 1, etc., that is, 1 first data field may be converted into 1 second data field, 1 first data field may be converted into a plurality of second data fields, or a plurality of first data fields may be converted into 1 second data field, etc., and a required conversion relationship may be created according to an implementation situation, which is not limited herein.

The conversion relationship is used to represent conversion processing required when the first data field is migrated to the second data field, including, for example, direct migration, migration of partial data in the first data field, merging multiple first data fields for migration, and the like, and is not limited herein.

The conversion relationship can be set as the required conversion relationship by adding a group of field relationships in the data migration setting interface by clicking the newly added corresponding relationship button in fig. 2; if the conversion relation needs to be modified, the corresponding first data field or second data field can be directly modified; if the conversion relation is not needed, a delete button can be directly clicked to delete the corresponding conversion relation, and the like.

Further, the created conversion relation can be stored, so that the first data field in the source data table can be migrated to other target databases by multiplexing the created conversion relation later.

Step S103, responding to the conversion operation triggered by the user, and reading the first data field in the source data table according to the conversion relation so as to realize the conversion and migration of the first data field to the second data field of the target data table of the target database.

After creating the conversion relationship, a user may click a start execution button in the data migration setting interface to trigger conversion migration of data, respond to a conversion operation triggered by the user, correspondingly read a first data field in the source data table, perform processing such as intercepting part of data in the first data field, merging a plurality of first data fields, directly acquiring the first data field, and the like on the first data field according to the conversion relationship in the data migration setting interface, migrate the processed data to a second data field corresponding to the target data table, and complete data migration.

The embodiment can be suitable for data migration between different databases, and a user only needs to set a conversion relation between a first data field in the source data table and a second data field in the target data table, so that data migration to the target database in the source data table can be realized, the data migration to different databases can be multiplexed, and the data migration efficiency is greatly improved.

According to the database migration method provided by the invention, visual data migration is provided through the data migration setting interface, the migration of the data in the source database to the target database can be rapidly completed by setting the conversion relation among the data, the data conversion code does not need to be manually compiled, the speed is higher, the convenience is higher, and the processing difficulty is reduced.

Example two

Fig. 3 is a flow chart of a database migration method according to another embodiment of the present invention, as shown in fig. 3, the method includes the following steps:

step S301, at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, are obtained based on a data migration setting interface.

The data migration setting interface may also obtain historical data input by the user, for example, the historical data input by the user is shown in a historical data drop-down box in fig. 2, if the user selects a certain historical data, the stored historical data is read to obtain a source data table, a target data table, and the like, and the newly added data in the source data table is migrated; or modify the historical data for other data migration, etc.

Step S302, creating a filtering condition for the source data table.

Considering that data in the source data table has already undergone partial data migration, or data migration is performed for specified data this time, the present embodiment also provides filtering condition setting. If the new filtering condition in fig. 2 is clicked, the filtering condition may be set in the data migration setting interface, where the filtering condition includes a filtering symbol and a filtering field of the first data field, where the filtering symbol includes various symbols such as a greater than number, an equal to number, a less than number, and an unequal number, and the filtering field may specify a certain first data field, and may form a filtering condition according to the filtering symbol and the filtering field, and screen out data that meets the filtering condition for migration through the filtering condition, or screen out data that meets the filtering condition as data that is not migrated this time, and so on. Taking the source database as the column cluster database as an example, if f-column qualifier type (type) of a certain column cluster is set to be equal to 1, data with type of 1 in f-column clusters meeting the filtering condition can be filtered, and migration is not performed.

Step S303, creating a conversion relationship between a first data field in at least one source data table and a second data field in a target data table.

The first data field in the column cluster database includes a row key, a column cluster, a column qualifier, etc., wherein the column cluster may contain a plurality of column qualifiers thereunder, each column qualifier having a respective corresponding column value, as shown in fig. 4. Further, to distinguish different versions of column qualifiers, each column qualifier may also include a temporal version, which may correspond to different column values, etc.

When creating the conversion relationship, the conversion relationship may include, for example, a field correspondence, a data type correspondence, data table primary key designation information, and the like.

Specifically, a corresponding relationship between the column cluster, the column qualifier and the field of the second data field is created, as shown in fig. 2, a corresponding relationship between the column cluster f, the source field (column qualifier) book _ id and the target field (second data field) book _ id is created. Further, considering that the data types of the column cluster database and the target database may be different, the conversion relationship includes a data type conversion corresponding relationship between the first data field and the second data field, for example, the first data field is int type, the second data field is bootean type, and the like, and the data type of the first data character may be converted correspondingly during data migration, so as to obtain a corresponding second data field for data migration.

During data migration, in order to facilitate operations such as management and data query on migrated data, when a conversion relationship is created, the second data field may be further specified and set as a unique primary key of the target data table, for example, the unique keys of the target fields book _ id, app _ id, book _ type, and user _ name in fig. 2 are set as yes, that is, the target fields are combined to be used as the primary key of the target data table, so that data of the target data table is conveniently searched subsequently. When the unique main key is set, one second data field can be set as the unique main key, or a plurality of second data fields can be set as the unique main key jointly, specifically, the data can be quickly searched according to the unique main key during query according to the expression condition of the target data.

When a source database is a column cluster database, and a first data field in the column cluster database comprises different fields of a row key, a column cluster and a column qualifier, the different fields can be subjected to different conversion processing according to different requirements when the first data field is migrated to a target database with different types. If the column cluster includes a plurality of column qualifiers, if it is necessary to migrate the column values of the plurality of column qualifiers into one second data field, the conversion relationship may be set as a column cluster merge relationship, and in fig. 2, the term of column cluster merge may be set as yes, if it is set as yes, the column values of all column qualifiers in the specified column cluster in the conversion relationship are merged into one item of data to be migrated into the second data field, and if the column cluster f is set as column cluster merge, the plurality of column qualifiers and the column value data in the column cluster f in fig. 4 are merged into one large field to be migrated into the corresponding second data field. At this time, only the column cluster, the target field and the target type in the conversion relationship may be set; alternatively, as shown in fig. 4, the row key includes multiple sets of continuous data, and the data uses a connector "-", the multiple sets of continuous data may correspond to different business meanings, respectively, and the multiple sets of continuous data of the row key in fig. 4 are name-type-status, that is, name is 1000i, type is TTT9112, and status is 20210205. When data are migrated, segmenting and intercepting row keys as required, selecting partial data as migration fields, setting row key interception relation in conversion relation, if key median of f-column cluster key row keys is specified as yes in fig. 2, namely acquiring data in the key row keys for migration, specifying an interceptor as "-", and specifying an intercepted group as a2 nd group, namely migrating type data TTT9112 to a second data field app _ id; or, for the row key, considering the requirement of service storage, for example, if the name is stored at the beginning of i, the name is stored only in the row key where i is located, and other row keys cannot be stored, so that the name is stored in a 1000i manner, and the names can be stored in different row keys in a scattered manner. A field inversion relationship is established in the conversion relationship, for example, in fig. 2, after the source field key2 is intercepted, the inversion is set, the name obtained by interception can be inverted, and then the name is migrated to the second data field of the target database, so that the user _ name stored in the target database can easily understand the meaning of the name. Further, the naming specification of the row key is generally classified by first characters, such as 0-9, "0 XXX", "1 XXX", "2 XXX", and the like, the row key can be accurately identified based on the naming specification, but sometimes a special condition occurs in the row key, such as i is used as the beginning of the first character, when field inversion is performed, in order to distinguish the i beginning first character from data of i0001 and the like of the name of the field inversion, a special key row key option is also set in fig. 2, a special key can be set therein as i to represent that the row key is a special row key, and the row key is distinguished from the naming specification, so that the row key can be distinguished when field inversion is performed, and error processing and the like are avoided.

Step S304, responding to the conversion operation triggered by the user, and reading the first data field in the source data table according to the conversion relation so as to realize the conversion and migration of the first data field to the second data field of the target data table of the target database.

When data migration is performed in response to a conversion operation triggered by a user, in order to improve migration processing efficiency, a multithreading manner may be adopted to read a first data field of a source data table, for example, a plurality of threads are used to read data of a column cluster of different row keys in parallel, for example, thread 1 reads row key 1, and thread 2 reads row key2 …, so as to improve reading efficiency of the first data field.

When the first data field is read, the configuration attributes of the execution equipment are considered, and the single data acquisition amount can be set by considering the attributes such as a cpu, a memory and the like, so that the problems of low operation efficiency and the like caused by once reading of too much data are avoided. The single data acquisition amount can be set according to implementation, as shown in fig. 2, such as setting a single query amount 800, that is, reading 800 pieces of data at a time. Furthermore, one thread reads the first data fields of the source data tables with corresponding quantity once according to the specified single data acquisition quantity, so that the execution speed is improved, and the whole operation effect is considered.

And performing corresponding conversion migration on the read first data field (here, the first data field meeting the filter condition is read according to the filter condition) according to the conversion relation. If the conversion relationship is the corresponding relationship between the column clusters and the column qualifiers in the first data field and the fields of the second data field, the column values corresponding to the column clusters and the column qualifiers can be directly read and obtained, and are converted according to the corresponding data types and transferred to the second data field; if the conversion relationship includes a column cluster merging relationship, obtaining a designated merged column cluster, merging each column qualifier included in the column cluster to obtain a merged field, as shown in fig. 4, adding a carename: a test card; desc: thirdly, performing primary filtration; origin: and 3, combining the fields into a large field, such as a json field format and the like, and migrating the obtained merged field to a second data field of the target data table. The format of the fields obtained by merging may be set according to implementation, and is not limited herein. If the conversion relation comprises a row key interception relation, an appointed row key of the source data table needs to be read, the row key is intercepted according to an appointed interceptor, and corresponding intercepted data is obtained according to an appointed interception group. If the row key in fig. 4 is intercepted according to the designated interceptor "-", the group 2 is intercepted, and intercepted data TTT9112 is obtained, and the intercepted data is migrated to the second data field app _ id of the target data table. And if the conversion relation comprises a field inversion relation, performing inversion processing on the specified first data field to obtain inverted data. The inversion process is to invert the original first data field, e.g., 1000i, to obtain inverted data i 0001. The designated first data field may be any column cluster or column qualifier, or may be a row key or intercepted data obtained by intercepting a row key. The resulting inverted data is migrated to a second data field of the target data table. As shown in fig. 2, the column cluster f-row key2 is first subjected to the intercepting process, the intercepted data after the intercepting process is subjected to the inverting process, and the obtained inverted data is migrated to the user _ name.

Optionally, in addition to migrating the data in the source database to the target database during data migration, fig. 2 also includes generating a first data field export in the source database as a specified data file, for example, specifying a text export when the target database is selected, and generating a format of the specified data file, such as a csv file, according to the conversion correspondence. The data in the source database of the text export can be imported into any target database (the target database is a database supporting file import). When a specified data file is imported, if the file is large, resource consumption during import can be affected, file size limitation can be set when the specified data file is generated, for example, files are exported in units of threads, and each thread exports and generates one specified data file, or the size of a single data file is set, and when the generated data file, such as 'name _1. cvs', reaches the size of the specified data file, a new file 'name _2. cvs' is exported to continue to store data, and the like.

Step S305, in response to the migration stop operation triggered by the user, stops the data migration processing from the source database to the target database.

In the data migration process, if a requirement for stopping data migration, such as an execution error, an incorrect operation, an execution failure, or the like, occurs, the operation entry provided in fig. 2 for the user to stop execution may stop the data migration processing from the current source database to the target database in response to the migration stop operation triggered by the user. If the currently selected text is exported, the generated specified data file can be deleted, and the problems of incomplete specified data file, incomplete data during import and the like are avoided.

According to the database migration method provided by the invention, the source database is supported to perform data migration to different target databases, the first data field of the source database is adapted, different conversion relations aiming at the first data field are created, and the data migration from the source database to the target database is realized. Furthermore, the data to be migrated can be screened through the filtering condition, and incremental migration of the data is realized. The first data field is read in a multithreading mode to improve the data migration speed, the specified single data acquisition amount can guarantee the normal execution during data migration, and the risks of downtime and the like caused by overlarge data amount are avoided.

EXAMPLE III

Fig. 5a is a flow chart of a database migration method according to another embodiment of the present invention, and as shown in fig. 5a, the method includes the following steps:

step S501, monitoring the data migration status of the source database and the target database in real time, and acquiring the current migrated data volume and/or the data volume to be migrated.

Step S502, displaying the current migrated data volume and/or the data volume to be migrated.

In the data migration process, the present embodiment monitors the data migration states of the source database and the target database in real time, specifically, monitors the source database and the target database, and acquires the migrated data volume that has been migrated and the unfinished data volume to be migrated in each database according to the migration processing.

In order to facilitate the user to know the data migration progress in real time, the user can be shown the current data migration progress by clicking the status button in fig. 2. As shown in FIG. 5b, the left side of the state display diagram shows the currently migrated data size, including the migrated data size of each row key, and the right side shows the data size to be migrated. The diagram shows the state after the execution is completed, the right-side data amount to be migrated is 0, and if the data amount to be migrated exists, the corresponding data amount is shown on the right-side.

According to the database migration method provided by the invention, the data migration states of the source database and the target database are monitored in real time, and the display of the real-time migration states is provided for a user, so that the user can know the data migration states conveniently.

Example four

Fig. 6a is a flow chart of a database migration method according to another embodiment of the present invention, as shown in fig. 6a, the method includes the following steps:

step S601, monitoring the data migration status of the source database and the target database in real time, and acquiring migration error information generated during the migration process.

Step S602, displaying the migration error information.

In the data migration process, the present embodiment monitors the data migration status of the source database and the target database in real time, and specifically monitors the migration error information generated in the migration process of the source database and the target database. The migration error information comprises a source database data reading error, a target database writing mismatch error and the like.

A user can display migration error information in a state display diagram by clicking a status button in fig. 2, as shown in fig. 6b, an error Key displays a source database reading error, a data reading error of a row Key is caused by an excessively large thread single data acquisition amount and the like; the error SQL displays a write error caused by the fact that the target database is not matched with the target database field when the target database is written, and the like.

According to the database migration method provided by the invention, the data migration states of the source database and the target database are monitored in real time, and the migration error information display is provided for a user in the migration process, so that the user can clearly know the data with the generated migration error, and the erroneous data can be conveniently migrated again.

EXAMPLE five

Fig. 7 is a flowchart illustrating a database migration method according to another embodiment of the present invention, and as shown in fig. 7, the method includes the following steps:

step S701, obtaining migration verification information input by a user in response to a verification operation triggered by the user.

Step S702, calculating to obtain data check quantity according to the migration check information, so as to perform sampling check on the consistency of the migrated data in the source database and the target database according to the data check quantity.

After the data migration is completed, the migrated data can be checked to ensure the consistency of the data migration. Specifically, as the verification ratio is set in fig. 2, the user may select an appropriate migration verification ratio, such as 1%, according to the implementation situation. The user clicks the start verification button in fig. 2, responds to the verification operation triggered by the user, obtains the migration verification information input by the user in the data migration setting interface, such as migration verification proportion information, single data acquisition amount and the like, and calculates the data verification amount according to the migration verification information. And if the migration check proportion information is multiplied by the single data acquisition amount, calculating to obtain the data check amount of the primary check. If the migration check proportion information is 1%, multiplying by the single data acquisition amount 800 to obtain a data check amount of 8 in one check. And sampling and checking the consistency of the migrated data in the source database and the target database according to the data checking quantity, namely, 8 pieces of data are extracted from the source database and the target database at a time to check the consistency of the data so as to ensure the accuracy of data migration.

Optionally, the user may also click the check state button in fig. 2, and display the check state information to the user, if the check result is obtained according to the sampling check, if the check result is data consistency, the check state is displayed to the user as data consistency; and if the verification result is that the data are inconsistent, displaying the data with the verification state of being inconsistent in the source database and the target database to a user so as to analyze problems and the like in the data migration.

According to the database migration method provided by the invention, data verification processing after data migration is provided, so that the data migration result can be conveniently verified, the consistency of data migration is detected, and the accuracy of data migration is ensured.

EXAMPLE six

The sixth embodiment of the present application provides a non-volatile computer storage medium, where the computer storage medium stores at least one executable instruction, and the computer executable instruction may execute the database migration method in any of the above method embodiments.

The executable instructions may be specifically configured to cause the processor to: acquiring at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, based on a data migration setting interface;

In an alternative embodiment, the source database comprises a column cluster database; the target database comprises a relational database; the first data field includes a row key, a column cluster, and/or a column qualifier.

In an alternative embodiment, the transforming further comprises: the field corresponding relation, the data type corresponding relation and/or the data table main key specifying information; the executable instructions further cause the processor to: creating a corresponding relation among the fields of the column cluster, the column qualifier and the second data field, and a data type conversion corresponding relation among the fields; the specified second data field is set to the unique primary key of the target data table for looking up data in the target data table.

In an alternative embodiment, converting the relationship comprises: a column cluster merging relationship; the executable instructions further cause the processor to: acquiring a column cluster appointed to be combined, and combining each column qualifier contained in the column cluster to obtain a combined field; and migrating the merged field to a second data field of the target data table.

In an alternative embodiment, converting the relationship comprises: intercepting the relation by a row key; the executable instructions further cause the processor to: reading an appointed row key of a source data table, and intercepting the row key according to an appointed interceptor to obtain intercepted data of an appointed intercepting group of the row key; migrating the intercepted data to a second data field of the target data table.

In an alternative embodiment, converting the relationship comprises: a field reversal relationship; the executable instructions further cause the processor to: reading a designated first data field of a source data table, and performing field inversion processing on the first data field to obtain inverted data; migrating the inverted data to a second data field of the target data table.

In an alternative embodiment, the executable instructions further cause the processor to: and reading a first data field of the source data table in a multithreading mode.

In an alternative embodiment, the executable instructions further cause the processor to: and reading a first data field of the source data table according to the specified single data acquisition amount.

In an alternative embodiment, the executable instructions further cause the processor to: creating a filter condition for the source data table; the filter condition includes a filter symbol and a filter field of the first data field.

In an alternative embodiment, the executable instructions further cause the processor to: monitoring the data migration state of a source database and a target database in real time, and acquiring the current migrated data volume and/or data volume to be migrated; and displaying the current migrated data volume and/or the data volume to be migrated.

In an alternative embodiment, the executable instructions further cause the processor to: monitoring the data migration state of a source database and a target database in real time, and acquiring migration error information generated in the migration process; the migration error information includes: a source database data reading error and/or a target database writing mismatch error; and displaying the migration error information.

In an alternative embodiment, the executable instructions further cause the processor to: responding to a verification operation triggered by a user, and acquiring migration verification information input by the user; the migration check information comprises migration check proportion information and/or single data acquisition quantity; and calculating to obtain data check quantity according to the migration check information so as to perform sampling check on the consistency of the migrated data in the source database and the target database according to the data check quantity.

In an alternative embodiment, the executable instructions further cause the processor to: exporting a first data field in a source data table to generate a specified data file; and the format of the specified data file is generated according to the conversion corresponding relation.

In an alternative embodiment, the executable instructions further cause the processor to: and stopping the data migration processing from the source database to the target database in response to the migration stopping operation triggered by the user.

EXAMPLE seven

Fig. 8 is a schematic structural diagram of an electronic device according to a seventh embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the electronic device.

As shown in fig. 8, the electronic device may include: a processor (processor)802, a Communications Interface 804, a memory 806, and a communication bus 808.

Wherein:

the processor 802, communication interface 804, and memory 806 communicate with one another via a communication bus 808.

A communication interface 804 for communicating with network elements of other devices, such as clients or other servers.

The processor 802 is configured to execute the program 810, and may specifically perform the relevant steps in the above-described database migration method embodiment.

In particular, the program 810 may include program code comprising computer operating instructions.

The processor 802 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement embodiments of the present invention. The server comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

The memory 806 stores a program 810. The memory 806 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 810 may be specifically configured to cause the processor 802 to perform the following operations: acquiring at least one source data table of a source database to be migrated and a target data table of a target database, which are input by a user, based on a data migration setting interface;

In an alternative embodiment, the transforming further comprises: the field corresponding relation, the data type corresponding relation and/or the data table main key specifying information; program 810 is for causing processor 802 to create a column cluster, a column qualifier, and field mapping of second data fields, and a data type conversion mapping between fields; the specified second data field is set to the unique primary key of the target data table for looking up data in the target data table.

In an alternative embodiment, converting the relationship comprises: a column cluster merging relationship; the program 810 is configured to enable the processor 802 to obtain a column cluster designated to be merged, and merge each column qualifier included in the column cluster to obtain a merged field; and migrating the merged field to a second data field of the target data table.

In an alternative embodiment, converting the relationship comprises: intercepting the relation by a row key; the program 810 is configured to enable the processor 802 to read a specified row key of the source data table, and perform interception processing on the row key according to a specified interceptor, to obtain intercepted data of a specified interception group of the row key; migrating the intercepted data to a second data field of the target data table.

In an alternative embodiment, converting the relationship comprises: a field reversal relationship; the program 810 is configured to enable the processor 802 to read a specified first data field of the source data table, and perform field inversion processing on the first data field to obtain inverted data; migrating the inverted data to a second data field of the target data table.

In an alternative embodiment, the program 810 is configured to cause the processor 802 to read a first data field of a source data table in a multi-threaded manner.

In an alternative embodiment, the program 810 is configured to cause the processor 802 to read the first data field of the source data table by a specified single data fetch.

In an alternative embodiment, the program 810 is for causing the processor 802 to create a filter term for the source data table; the filter condition includes a filter symbol and a filter field of the first data field.

In an alternative embodiment, the program 810 is configured to enable the processor 802 to monitor the data migration status of the source database and the target database in real time, and obtain the current migrated data size and/or the data size to be migrated; and displaying the current migrated data volume and/or the data volume to be migrated.

In an alternative embodiment, the program 810 is configured to enable the processor 802 to monitor the data migration status of the source database and the target database in real time, and obtain the migration error information generated in the migration process; the migration error information includes: a source database data reading error and/or a target database writing mismatch error; and displaying the migration error information.

In an alternative embodiment, the program 810 is configured to enable the processor 802 to obtain migration verification information input by a user in response to a user-triggered verification operation; the migration check information comprises migration check proportion information and/or single data acquisition quantity; and calculating to obtain data check quantity according to the migration check information so as to perform sampling check on the consistency of the migrated data in the source database and the target database according to the data check quantity.

In an alternative embodiment, program 810 is configured to cause processor 802 to generate a first data field export in a source data table as a specified data file; and the format of the specified data file is generated according to the conversion corresponding relation.

In an alternative embodiment, the program 810 is configured to cause the processor 802 to stop the data migration process of the source database to the target database in response to a user-triggered migration stop operation.

For specific implementation of each step in the program 810, reference may be made to the corresponding description in the corresponding step in the foregoing database migration embodiment, which is not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.

According to the scheme provided by the embodiment, visual data migration is provided through the data migration setting interface, the migration of data in the source database to the target database can be completed quickly by setting the conversion relation among the data, the data conversion code does not need to be written manually, the speed is higher, the convenience is higher, and the processing difficulty is reduced.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

The invention discloses: A1. a database migration method, comprising:

creating a conversion relationship between a first data field in the at least one source data table and a second data field in the target data table;

and responding to conversion operation triggered by a user, and reading a first data field in the source data table according to the conversion relation so as to realize conversion and migration of the first data field to a second data field of a target data table of the target database.

A2. The method of a1, wherein the source database comprises a column cluster database; the target database comprises a relational database; the first data field includes a row key, a column cluster, and/or a column qualifier.

A3. The method of a2, wherein the transforming further comprises: the field corresponding relation, the data type corresponding relation and/or the data table main key specifying information;

the creating a conversion relationship between a first data field in the at least one source data table and a second data field in the target data table further comprises:

creating a corresponding relation among the fields of the column clusters, the column qualifiers and the second data fields and a data type conversion corresponding relation among the fields;

and setting the specified second data field as a unique primary key of the target data table so as to search the data in the target data table.

A4. The method of a2, wherein the translation relationship includes: a column cluster merging relationship;

the reading the first data field in the source data table according to the conversion relationship to realize the conversion migration of the first data field to the second data field of the target data table of the target database further comprises:

acquiring a column cluster appointed to be combined, and combining each column qualifier contained in the column cluster to obtain a combined field;

and migrating the merged field to a second data field of the target data table.

A5. The method of a2, wherein the translation relationship includes: intercepting the relation by a row key;

reading an appointed row key of the source data table, and intercepting the row key according to an appointed interceptor to obtain intercepted data of an appointed intercepting group of the row key;

migrating the intercepted data to a second data field of a target data table.

A6. The method of a2, wherein the translation relationship includes: a field reversal relationship;

reading a specified first data field of the source data table, and performing field inversion processing on the first data field to obtain inverted data;

migrating the inverted data to a second data field of a target data table.

A7. The method as in A1-A6, wherein the reading the first data field in the source data table is specifically: and reading a first data field of the source data table in a multithreading mode.

A8. The method as in A1-A7, wherein the reading the first data field in the source data table is specifically: and reading a first data field of the source data table according to the specified single data acquisition amount.

A9. The method as in A1-A8, wherein the method further comprises:

creating a filter condition for the source data table; the filter condition includes a filter symbol and a filter field of the first data field.

A10. The method as in A1-A9, wherein the method further comprises:

monitoring the data migration state of a source database and a target database in real time, and acquiring the current migrated data volume and/or data volume to be migrated;

and displaying the current migrated data volume and/or the data volume to be migrated.

A11. The method as in A1-A10, wherein the method further comprises:

monitoring the data migration state of a source database and a target database in real time, and acquiring migration error information generated in the migration process; the migration error information includes: a source database data reading error and/or a target database writing mismatch error;

and displaying the migration error information.

A12. The method as in A1-A11, wherein the method further comprises:

responding to a verification operation triggered by a user, and acquiring migration verification information input by the user; the migration check information comprises migration check proportion information and/or single data acquisition amount;

and calculating to obtain data check quantity according to the migration check information so as to perform sampling check on the consistency of the migrated data in the source database and the target database according to the data check quantity.

A13. The method of any one of a1-a12, wherein the method further comprises:

exporting a first data field in the source data table to generate a specified data file; and the format of the specified data file is generated according to the conversion corresponding relation.

A14. The method of any one of a1-a13, wherein the method further comprises:

and responding to the migration stopping operation triggered by the user, and stopping the data migration processing from the source database to the target database.

The invention also discloses: B15. an electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is configured to store at least one executable instruction that causes the processor to:

B16. The electronic device of B15, the source database comprising a column cluster database; the target database comprises a relational database; the first data field includes a row key, a column cluster, and/or a column qualifier.

B17. The electronic device of B16, the translating relationship further comprising: the field corresponding relation, the data type corresponding relation and/or the data table main key specifying information;

the executable instructions further cause the processor to:

B18. The electronic device of B16, the transforming relationship comprising: a column cluster merging relationship;

the executable instructions further cause the processor to:

and migrating the merged field to a second data field of the target data table.

B19. The electronic device of B16, the transforming relationship comprising: intercepting the relation by a row key;

the executable instructions further cause the processor to:

migrating the intercepted data to a second data field of a target data table.

B20. The electronic device of B16, the transforming relationship comprising: a field reversal relationship;

the executable instructions further cause the processor to:

migrating the inverted data to a second data field of a target data table.

B21. The electronic device of B15-B20, the executable instructions further cause the processor to:

and reading a first data field of the source data table in a multithreading mode.

B22. The electronic device of B15-B21, the executable instructions further cause the processor to:

and reading a first data field of the source data table according to the specified single data acquisition amount.

B23. The electronic device of B15-B22, the executable instructions further cause the processor to:

B24. The electronic device of B15-B23, the executable instructions further cause the processor to:

B25. The electronic device of B15-B24, the executable instructions further cause the processor to:

and displaying the migration error information.

B26. The electronic device of B15-B25, the executable instructions further cause the processor to:

B27. The electronic device of any of B15-B26, the executable instructions further cause the processor to:

B28. The electronic device of any of B15-B27, the executable instructions further cause the processor to:

The invention also discloses: C29. a computer storage medium having stored therein at least one executable instruction that causes a processor to perform operations corresponding to the database migration method of any one of a1-a 14.

Claims

1. A database migration method, comprising:

2. The method of claim 1, wherein the source database comprises a column cluster database; the target database comprises a relational database; the first data field includes a row key, a column cluster, and/or a column qualifier.

3. The method of claim 2, wherein the translating further comprises: the field corresponding relation, the data type corresponding relation and/or the data table main key specifying information;

4. The method of claim 2, wherein the translating relationship comprises: a column cluster merging relationship;

and migrating the merged field to a second data field of the target data table.

5. The method of claim 2, wherein the translating relationship comprises: intercepting the relation by a row key;

migrating the intercepted data to a second data field of a target data table.

6. The method of claim 2, wherein the translating relationship comprises: a field reversal relationship;

migrating the inverted data to a second data field of a target data table.

7. The method according to claims 1-6, wherein the reading of the first data field in the source data table is specifically: and reading a first data field of the source data table in a multithreading mode.

8. The method according to claims 1-7, wherein the reading of the first data field in the source data table is specifically: and reading a first data field of the source data table according to the specified single data acquisition amount.

9. An electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

10. A computer storage medium having stored therein at least one executable instruction that causes a processor to perform operations corresponding to the database migration method of any one of claims 1-8.