CN113312331A

CN113312331A - Data migration method, device, system, electronic equipment and computer readable medium

Info

Publication number: CN113312331A
Application number: CN202010251758.8A
Authority: CN
Inventors: 徐世才
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-04-01
Filing date: 2020-04-01
Publication date: 2021-08-27

Abstract

The embodiment of the application provides a data migration method, a data migration device, a data migration system, electronic equipment and a computer readable medium, and relates to the field of databases. Wherein the method comprises the following steps: determining splitting suggestion data of user data to be migrated from the source relational database to the target distributed relational database based on the characteristic data of the user data to be migrated in the source relational database; establishing a target database for storing user data in a target distributed relational database based on the splitting suggestion data; migrating user data from the source relational database to a target repository in the target distributed relational database. By the method and the device, the user data can be rapidly migrated from the source relational database to the target distributed relational database.

Description

Data migration method, device, system, electronic equipment and computer readable medium

Technical Field

The embodiment of the application relates to the field of databases, in particular to a data migration method, a data migration device, a data migration system, electronic equipment and a computer readable medium.

Background

The vigorous development of the internet has higher and higher requirements on the performance and storage of databases, and although various non-relational databases are continuously updated to solve the query problem of data storage in many specific scenes, the relational database is still the mainstream of the database industry due to the advantages of excellent performance, comprehensive functions, stability, easy use and the like. However, with the explosive increase of data volume, the traditional single-machine relational database has difficulty in meeting the requirements in storage and performance, and various distributed relational databases are produced.

In an actual application scenario, a user usually uses a stand-alone relational database, and to switch to a distributed relational database, user data needs to be migrated. The existing open-source or charged data migration tools are all directed at the data migration scene of a general single-machine relational database, and do not specially process the scene that user data is migrated from the single-machine relational database to a distributed relational database. In particular, data migration from one type of stand-alone relational database to another type of stand-alone relational database, such as migrating user data from Oracle to Mysql. Data migration from one stand-alone relational database instance to another, such as migrating user data from one Mysql instance to another Mysql instance. Because special processing is not specially performed on a scene in which user data is migrated from the single-machine relational database to the distributed relational database, a user needs to select a splitting scheme in advance and build a database and a table by himself. However, the threshold for using a distributed relational database is too high for the user. The basis of using the distributed relational database is to split data, and mainly comprises database division and table division. For a data table to be split, a user usually needs three steps to split the data table: and selecting a splitting key, and drawing up a splitting rule to realize splitting. To accomplish these steps, the user needs to have a fair understanding of the database principles and distributed database technology, resulting in a very slow migration of user data from a stand-alone relational database to a distributed relational database.

Therefore, how to rapidly migrate user data from the relational database to the distributed relational database becomes a technical problem to be solved urgently at present.

Disclosure of Invention

An object of the embodiments of the present application is to provide a data migration method, apparatus, system, electronic device, and computer readable medium, which are used to solve the technical problem in the prior art of how to quickly migrate user data from a stand-alone relational database to a distributed relational database.

According to a first aspect of embodiments of the present application, a data migration method is provided. The method comprises the following steps: determining splitting suggestion data of user data to be migrated from a source relational database to a target distributed relational database based on feature data of the user data to be migrated in the source relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database.

According to a second aspect of embodiments of the present application, a data migration method is provided. The method comprises the following steps: receiving an operation for instructing migration of user data in a source relational database to a target distributed relational database; and sending a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

According to a third aspect of embodiments of the present application, a data migration system is provided. The system comprises: the client is used for receiving an operation for indicating that user data in a source relational database is migrated to a target distributed relational database, and sending a data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation; and the data migration tool is used for receiving the data migration request sent by the client and migrating the user data in the source relational database to the target distributed relational database based on the data migration request.

According to a fourth aspect of embodiments of the present application, there is provided a data migration apparatus. The device comprises: the determining module is used for determining splitting suggestion data of the user data transferred from the source relational database to the target distributed relational database based on the characteristic data of the user data to be transferred in the source relational database; the establishing module is used for establishing a target database for storing the user data in the target distributed relational database based on the splitting suggestion data; a migration module to migrate the user data from the source relational database to the target library in the target distributed relational database.

According to a fifth aspect of embodiments of the present application, there is provided a data migration apparatus. The device comprises: a third receiving module, configured to receive an operation for instructing migration of user data in a source relational database to a target distributed relational database; and the sending module is used for sending a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

According to a sixth aspect of embodiments of the present application, there is provided an electronic apparatus, including: one or more processors; a computer readable medium configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data migration method of the first or second aspect as described in the above embodiments.

According to a seventh aspect of embodiments of the present application, there is provided a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the data migration method as described in the first or second aspect of the embodiments above.

According to the data migration scheme provided by the embodiment of the application, based on the characteristic data of the user data to be migrated in the source relational database, the splitting suggestion data for migrating the user data from the source relational database to the target distributed relational database is determined, based on the splitting suggestion data, the target database for storing the user data is established in the target distributed relational database, then the user data is migrated from the source relational database to the target database in the target distributed relational database, compared with the existing other modes, based on the characteristic data of the user data to be migrated in the source relational database, the splitting suggestion data for migrating the user data from the source relational database to the target distributed relational database is determined, and based on the splitting suggestion data, the target database for storing the user data is established in the target distributed relational database, the method can hide a large amount of repeated work of the user, greatly reduces the use threshold of the distributed relational database by the user, can quickly migrate the user data from the source relational database to the target distributed relational database, and realizes the quick and integrated experience of data migration from the source relational database to the target distributed relational database.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1A is a schematic diagram of a database server provided in accordance with an embodiment of the present application;

FIG. 1B is a flowchart illustrating steps of a data migration method according to an embodiment of the present disclosure;

fig. 1C is a schematic diagram of a determination process of the splitting recommendation data according to an embodiment of the present application;

FIG. 2 is a flowchart illustrating steps of a data migration method according to a second embodiment of the present application;

FIG. 3 is a flowchart illustrating steps of a data migration method according to a third embodiment of the present application;

FIG. 4 is a diagram illustrating a data migration system according to a fourth embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data migration apparatus according to a fifth embodiment of the present application;

FIG. 6 is a schematic structural diagram of a data migration apparatus according to a sixth embodiment of the present application;

FIG. 7 is a schematic structural diagram of a data migration apparatus according to a seventh embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device in an eighth embodiment of the present application;

fig. 9 is a hardware configuration of an electronic device according to a ninth embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Before describing the data migration method provided by the embodiment of the present application in detail, technical terms related to the data migration method provided by the embodiment of the present application are described in detail.

A server may be understood as a specific physical machine on which multiple database instances may be deployed. As shown in FIG. 1A, two database instances, database instance Port1 and database instance Port2, are deployed on a database server. Multiple databases may be deployed on a database instance. As shown in FIG. 1A, database db0, database db1, database db2 and database db3 are deployed on database instance Port 1. Database db0, database db1, database db2 and database db3 are deployed on database instance Port 2. The database is a logical organization form of database tables, and a database can contain a plurality of data tables. For example, database db1, deployed on database instance Port1, contains data tables 0, 1, 2, 3, 4, 5, 6, and 7. The data table can be understood as a data storage unit, the index structure is B + Tree, generally, the capacity of a single table is not too large, the number of layers of the B + Tree is higher due to the fact that the capacity of the single table is too large, and data reading and writing on the B + Tree are relatively slow.

For a service program, the object of connection management is a database (containing one or more data tables); the sub-database and sub-table are referred to as database and data table. The splitting of the service program to the data only needs to sense the database and the data table, and as for how many database instances are located, the service program is not concerned about how many machines are deployed. I.e. databases and data tables, are a category of so-called logical libraries.

For actual deployment, the concern is how many machines are deployed, how many database instances are deployed per machine, which is also the cost of the resources actually consumed. I.e., machine and database instances, are a category of what is commonly referred to as physical libraries.

For business data with relatively stable growth, the business data is generally split directly according to the encountered bottleneck problem, and the business data is dispersed to a certain data table, database instance or machine, at this time, the logical library and the physical library are basically consistent, and further splitting is not needed or possible.

For traffic data that grows relatively fast (or is expected to grow faster), the bottleneck problem is typically evaluated in the service program to split by the amount of growth possible over a period of time in the future. In actual deployment, not as many entity machines and database instances are actually deployed, so that the logical library and the physical library have a certain mapping relationship (the logical library is larger than the physical library). When the service data is increased to a certain degree, under the condition that the program logic is not changed, the capacity can be expanded by splitting the physical library until the logic library and the physical library are consistent, namely the maximum bearing bottleneck of the service program is achieved.

Referring to fig. 1B, a flowchart illustrating steps of a data migration method according to a first embodiment of the present application is shown.

The present embodiment describes a data migration method provided in the present embodiment from the perspective of a data migration tool. Specifically, the data migration method provided in this embodiment includes the following steps:

in step S101, based on the feature data of the user data to be migrated in the source relational database, the suggested splitting data for migrating the user data from the source relational database to the target distributed relational database is determined.

In the embodiment of the present application, a relational database may be understood as a database that organizes data by using a relational model, and stores data in the form of rows and columns so as to be easily understood by a user. A user retrieves data in a database by a query, which is an executable code that defines certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and relations between them. The source relational database can be a source single-machine relational database, a source relational database of an offline data center, a source distributed relational database and the like. A stand-alone database is understood to be a database that can only run on a stand-alone computer and does not provide network functionality. The source stand-alone relational database may be understood to be a source relational database that runs on a stand-alone. The user data to be migrated may be commodity order data, commodity transaction data, and the like of the user. The distributed relational database may be understood to be a middleware-based distributed relational database system service provided. The distributed relational database can be used for constructing a database system supporting mass data storage and access based on a transverse expansion mode of a common server, so that infinite capacity expansion and elastic expansion are realized. Compared with a single-machine relational database, the distributed relational database provides a higher-specification storage query rate, and meets the continuously-increased mass data storage requirement of a user and the continuously-increased service request pressure. The target distributed relational database may be understood as a middleware-based target distributed relational database system service provided. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In an embodiment of the present application, the feature data of the user data to be migrated includes at least one of: the total amount of the user data in the source relational database, the amount of data in a data table for storing the user data in the source relational database, and the access characteristic data for the user data in the source relational database in a historical period. The split recommendation data includes at least one of: the suggested number of sub-libraries obtained by splitting the source relational database, the suggested number of sub-tables obtained by splitting the data tables in the source relational database, and the suggested split key for the data tables. The split key may be understood as a split field of the data table, for example, a user name split field, a user identification field, and the like. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, before determining that the user data is migrated from the source relational database to the split proposal data of the target distributed relational database, the method further comprises: receiving an account and a password of the source relational database input by a user at a client; and acquiring the total data amount of the user data and the data amount of the data table from the source relational database based on the account and the password of the source relational database. The account may be a read-only account of the source relational database, and the password may be a password of the read-only account of the source relational database. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, before determining that the user data is migrated from the source relational database to the split proposal data of the target distributed relational database, the method further comprises: receiving access authority data aiming at the log file of the source relational database, which is input by a user at a client; and extracting access characteristic data aiming at the user data in the historical time period from the log file based on the access authority data, wherein the log file is used for recording SQL statements for updating the user data in the source relational database. Wherein the access right data can be understood as data of log files of the source relational database which are authorized to be accessed by the data migration tool. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the access characteristic data for the user data in the historical time period is extracted from the log file, an SQL statement for updating the user data in the historical time period is extracted from the log file; and performing feature extraction on the SQL statements updating the user data in the historical time period to obtain access feature data aiming at the user data in the historical time period. Access characteristic data for the user data over the historical period of time includes at least one of: the number of the SQL statements updated for the user data in the historical time period, the types of the SQL statements updated for the user data in the historical time period, and the number of times of occurrence of the same field name in the SQL statements updated for the user data in the historical time period. The log file may be a log file in a binary format, and is used to record the SQL statements updated by the user on the user data and your rows in the relational database, for example, the SQL statements changing the database table and changing the content are both recorded in the log file in the binary format, but the query on the content such as the base table is not recorded. The log file in binary format is mainly used for master-slave copy of the relational database and incremental recovery of data. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when determining splitting recommendation data for migrating user data from a source relational database to a target distributed relational database based on feature data of the user data to be migrated in the source relational database, determining a recommendation number of split banks obtained by splitting the source relational database based on a total data amount of the user data in the source relational database and enriched data of the user data at a future time; determining a suggested number of sub-tables obtained by splitting the data table based on a data amount of the data table for storing the user data in the source relational database; determining a suggested split key for the data table based on access characteristic data for the user data in the source relational database over a historical period of time. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when determining the suggested number of segments obtained by splitting the source relational database based on the total amount of data of the user data in the source relational database and the potentialization data of the user data at the future time, adding the total amount of data of the user data in the source relational database and the potentialization data of the user data at the future time to obtain the total amount of data of the user data at the future time in the source relational database; and dividing the total data amount of the user data in the source relational database at the future time by the data storage capacity of the sub-databases to obtain the suggested number of the sub-databases obtained by splitting the source relational database. Wherein the incremental data of the user data at the future time can predict the data volume of the user data increasing at the future time for the user. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when the suggested number of the partial tables obtained by splitting the data table is determined based on the data amount of the data table used for storing the user data in the source relational database, the suggested number of the partial tables obtained by splitting the data table is obtained by dividing the data amount of the data table used for storing the user data in the source relational database by the data storage capacity of the partial tables. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when determining a proposed split key for the data table based on access characteristic data for the user data in the source relational database over a historical period of time, the proposed split key for the data table is determined based on the number of occurrences of the same field name in the same type of SQL statement that is updated for the user data stored in the data table over the historical period of time. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, when determining the proposed split key for the data table, if the type of the SQL statement is a select type, the more times the same field name appears in the SQL statement of the type updated for the user data stored in the data table within the historical time period, the greater the possibility that the field name is the proposed split key for the data table. If the type of the SQL statement is an update (update) type, the less the number of times the same field name appears in the SQL statement of the type updated for the user data stored in the data table within the historical time period, the greater the possibility that the field name is a split key suggested for the data table. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, as shown in fig. 1C, the total amount of data of the user data in the source relational database and the potentialization data of the user data at the future time are added to obtain the total amount of data of the user data in the source relational database at the future time, and the total amount of data of the user data in the source relational database at the future time is divided by the data storage capacity of the sub-banks to obtain the suggested number of sub-banks obtained by splitting the source relational database. Then, the data amount of the data table for storing the user data in the source relational database is divided by the data storage capacity of the sub-table to obtain the suggested number of sub-tables obtained by splitting the data table. And finally, determining a suggested split key for the data table based on the occurrence times of the same field name in the same type of SQL statement which is updated aiming at the user data stored in the data table in the historical time period. After obtaining the proposed number of sublots, the proposed number of sublists, and the proposed split key for the data table, the final split proposed data can be obtained. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S102, a target library for storing the user data is established in the target distributed relational database based on the splitting proposal data.

In the embodiment of the application, the target library comprises a target logical library and a target physical library having a mapping relation with the target logical library. The category of logical libraries is commonly referred to as databases and data tables. The category of physical libraries is commonly referred to as machine and database instances. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when a target repository for storing the user data is established in the target distributed relational database based on the splitting recommendation data, the target logical repository and the target physical repository for storing the user data are established in the target distributed relational database based on a recommended number of repositories obtained by splitting the source relational database, a recommended number of tables obtained by splitting a data table in the source relational database, and a split key recommended for the data table. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S103, migrating the user data from the source relational database to the target repository in the target distributed relational database.

In some optional embodiments, after migrating the user data from the source relational database to the target repository in the target distributed relational database, the method further comprises: acquiring SQL statements used for updating the user data in the migration time period of the user data from a log file of the source relational database, wherein the log file is used for recording the SQL statements used for updating the user data; and executing SQL sentences used for updating the user data in the migration time period of the user data aiming at the target library in the target distributed relational database. Wherein, the log file can be a log file in binary format, and the migration time period can be understood as the time period from the beginning of user data migration to the end of user data migration. Therefore, the migration of the user full data and the user incremental data in the source relational database can be realized, and the seamless connection of the user data from the source relational database to the target distributed relational database can be ensured. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

By the data migration method provided by the embodiment of the application, based on the characteristic data of the user data to be migrated in the source relational database, the splitting suggestion data for migrating the user data from the source relational database to the target distributed relational database is determined, based on the splitting suggestion data, the target database for storing the user data is established in the target distributed relational database, and then the user data is migrated from the source relational database to the target database in the target distributed relational database, compared with the existing other modes, based on the characteristic data of the user data to be migrated in the source relational database, the splitting suggestion data for migrating the user data from the source relational database to the target distributed relational database is determined, and based on the splitting suggestion data, the target database for storing the user data is established in the target distributed relational database, the method can hide a large amount of repeated work of the user, greatly reduces the use threshold of the distributed relational database by the user, can quickly migrate the user data from the source relational database to the target distributed relational database, and realizes the quick and integrated experience of data migration from the source relational database to the target distributed relational database.

The data migration method of the present embodiment may be performed by any suitable device having data processing capabilities, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.

Referring to fig. 2, a flowchart of steps of a data migration method according to a second embodiment of the present application is shown.

The present embodiment explains the data migration method provided in the present embodiment from the perspective of the client. Specifically, the data migration method provided in this embodiment includes the following steps:

in step S201, an operation is received for instructing migration of user data in a source relational database to a target distributed relational database.

In some optional embodiments, upon receiving an operation to instruct migration of user data in a source relational database to a target distributed relational database, receiving an operation for a data migration control exposed in a page of a client, the operation to instruct migration of user data in the source relational database to the target distributed relational database. The operation for the data migration control shown in the page of the client may be a click operation, a long-time press operation, or a drag operation for the data migration control shown in the page of the client. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, prior to the receiving operation for the data migration control exposed in the page of the client, the method further comprises: receiving a selection operation for a migration destination of the user data shown in a page of the client; and refreshing the page of the client based on the selection operation so as to enable the page of the client to display the data migration control. Thus, the user data in the source relational database can be migrated by the selection operation for the migration destination of the user data shown in the page of the client. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In a specific example, the migration destination of the user data may be a target distributed relational database or a target public cloud. And when the migration destination of the user data selected by the user in the page of the client is the target distributed relational database, refreshing the page of the client so as to enable the page of the client to display the data migration control. The data migration control is used for indicating the migration of the user data in the source relational database to the target distributed relational database under the condition of being operated. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S202, based on the operation, a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database is sent to a data migration tool, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

In some optional embodiments, after sending, to a data migration tool, a data migration request for requesting migration of user data in the source relational database to the target distributed relational database based on the operation, the method further includes: receiving a data migration response which is returned by the data migration tool based on the data migration request and carries a data migration completion message; and switching the source relational database to the target distributed relational database based on the data migration completion message carried in the data migration response. Therefore, the source relational database can be switched to the target distributed relational database through the data migration completion message carried in the data migration response. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

By the data migration method provided by the embodiment of the application, the operation for instructing the migration of the user data in the source relational database to the target distributed relational database is received, and based on the operation, a data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database is sent to the data migration tool, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request, compared with other existing methods, the operation for instructing the migration of the user data in the source relational database to the target distributed relational database is received, and based on the operation, a data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database is sent to the data migration tool, and the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request, and can automatically migrate the user data in the source relational database to the target distributed relational database.

Referring to fig. 3, a flowchart of steps of a data migration method in the third embodiment of the present application is shown.

In this embodiment, the data migration method provided in this embodiment is described from the perspective of interaction between a client and a data migration tool. Specifically, the data migration method provided in this embodiment includes the following steps:

in step S301, an operation is received for instructing migration of user data in a source relational database to a target distributed relational database.

Since the embodiment of step S301 is similar to that of step S201, it is not described herein again.

In step S302, based on the operation, a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database is sent to a data migration tool.

Since the embodiment of step S302 is similar to that of step S202, it is not repeated herein.

In step S303, the data migration request sent by the client is received.

In the embodiment of the application, the data migration tool receives the data migration request sent by the client. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In step S304, based on the data migration request, migrating the user data in the source relational database to the target distributed relational database.

In this embodiment of the application, the data migration tool migrates user data in the source relational database to the target distributed relational database based on the data migration request. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

In some optional embodiments, when migrating user data in the source relational database to the target distributed relational database, determining split suggestion data for migrating the user data from the source relational database to the target distributed relational database based on feature data of the user data to be migrated in the source relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.

By the data migration method provided by the embodiment of the application, the client receives the operation for instructing the migration of the user data in the source relational database to the target distributed relational database, and sends the data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database to the data migration tool based on the operation, the data migration tool receives the data migration request sent by the client, and migrates the user data in the source relational database to the target distributed relational database based on the data migration request, compared with other existing methods, the client receives the operation for instructing the migration of the user data in the source relational database to the target distributed relational database, and sends the data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database to the data migration tool based on the operation, the data migration tool receives a data migration request sent by a client, migrates user data in the source relational database to the target distributed relational database based on the data migration request, and can automatically migrate the user data in the source relational database to the target distributed relational database.

Referring to fig. 4, a schematic diagram of a data migration system in the fourth embodiment of the present application is shown.

The data migration system provided by the embodiment comprises: the client is used for receiving an operation for indicating that user data in a source relational database is migrated to a target distributed relational database, and sending a data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation; and the data migration tool is used for receiving the data migration request sent by the client and migrating the user data in the source relational database to the target distributed relational database based on the data migration request.

Optionally, the data migration tool is specifically configured to determine, based on feature data of user data to be migrated in a source relational database, split suggestion data for migrating the user data from the source relational database to a target distributed relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database.

Referring to fig. 4, in order to implement the data migration method provided in the embodiment of the present application, a structural diagram of a data migration system may include a data migration tool and a client in a terminal device a, and it should be understood that the data migration tool and the client in the terminal device a presented in fig. 4 are only exemplary illustrations and are not limited to implementation forms of the data migration tool and the client in the terminal device a.

In practical application, the data migration tool and the terminal device a may be connected by a wired or wireless network, and specifically, the data migration tool and the terminal device a may be connected by a mobile network such as GSM, GPRS, LTE, or the like, or may be connected by a bluetooth, WIFI, infrared, or the like.

The data migration tool may be a data migration tool arranged in a service device for providing a service by a user, and specifically may be an independent application service device.

The terminal device a may be a user-oriented terminal capable of interacting with a user, such as a mobile phone, a notebook, a computer, an iPad, an intelligent audio, and the like, and may be various self-service terminals, such as self-service machines in places such as hospitals, banks, stations, and the like, and in addition, the terminal device a may also be an intelligent machine supporting interaction, such as a chat robot, a floor sweeping robot, a meal ordering service robot, and the like. The product type and the physical form of the terminal equipment are not limited, and the terminal equipment needs to have an interactive function and can be realized by installing interactive application programs such as database and the like.

When data migration is performed, a client in the terminal device a may send a data migration request to the data migration tool through a network. And the data migration tool receives a data migration request sent by a client in the terminal equipment A, and migrates user data in the source relational database to the target distributed relational database based on the data migration request. Therefore, the data migration method provided by the embodiment of the present application may be executed by a data migration tool, and the specific implementation process may refer to the description of the first method embodiment.

With the data migration system provided in the embodiment of the present application, the client receives an operation for instructing migration of user data in the source relational database to the target distributed relational database, and based on the operation, sends a data migration request for requesting migration of user data in the source relational database to the target distributed relational database to the data migration tool, the data migration tool receives a data migration request sent by the client, and based on the data migration request, migrates user data in the source relational database to the target distributed relational database, and compared with other existing approaches, the client receives an operation for instructing migration of user data in the source relational database to the target distributed relational database, and based on the operation, sends a data migration request for requesting migration of user data in the source relational database to the target distributed relational database to the data migration tool, the data migration tool receives a data migration request sent by a client, migrates user data in the source relational database to the target distributed relational database based on the data migration request, and can automatically migrate the user data in the source relational database to the target distributed relational database.

The data migration apparatus of this embodiment is configured to implement the corresponding data migration method in the foregoing multiple method embodiments, and details are not repeated here.

Referring to fig. 5, a schematic structural diagram of a data migration apparatus in the fifth embodiment of the present application is shown.

The data migration apparatus provided in this embodiment includes: a determining module 501, configured to determine, based on feature data of user data to be migrated in a source relational database, split suggestion data for migrating the user data from the source relational database to a target distributed relational database; an establishing module 502, configured to establish a target library for storing the user data in the target distributed relational database based on the splitting recommendation data; a migration module 503, configured to migrate the user data from the source relational database to the target library in the target distributed relational database.

The data migration apparatus of this embodiment is configured to implement the corresponding data migration method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.

Referring to fig. 6, a schematic structural diagram of a data migration apparatus in a sixth embodiment of the present application is shown.

The data migration apparatus provided in this embodiment includes: a determining module 605, configured to determine, based on feature data of user data to be migrated in a source relational database, split suggestion data for migrating the user data from the source relational database to a target distributed relational database; an establishing module 606, configured to establish a target library for storing the user data in the target distributed relational database based on the splitting recommendation data; a migration module 607, configured to migrate the user data from the source relational database to the target repository in the target distributed relational database.

Optionally, the characteristic data of the user data to be migrated includes at least one of: the total amount of the user data in the source relational database, the amount of data in a data table for storing the user data in the source relational database, and the access characteristic data for the user data in the source relational database in a historical period.

Optionally, before the determining module 605, the apparatus further includes: a first receiving module 601, configured to receive an account and a password of the source relational database, which are input by a user at a client; a first obtaining module 602, configured to obtain, based on the account and the password of the source relational database, a total data amount of the user data and a data amount of the data table from the source relational database.

Optionally, before the determining module 605, the apparatus further includes: a second receiving module 603, configured to receive access right data, which is input by a user at a client and is for a log file of the source relational database; an extracting module 604, configured to extract, based on the access permission data, access characteristic data for the user data in the historical time period from the log file, where the log file is used to record an SQL statement that updates the user data in the source relational database.

Optionally, the access characteristic data for the user data over the historical period of time comprises at least one of: the number of the SQL statements updated for the user data in the historical time period, the types of the SQL statements updated for the user data in the historical time period, and the number of times of occurrence of the same field name in the SQL statements updated for the user data in the historical time period.

Optionally, the split recommendation data comprises at least one of: the suggested number of sub-libraries obtained by splitting the source relational database, the suggested number of sub-tables obtained by splitting the data tables in the source relational database, and the suggested split key for the data tables.

Optionally, the determining module 605 includes: a first determining sub-module 6051, configured to determine, based on a total amount of the user data in the source relational database and the incremental data of the user data at a future time, a suggested number of sub-libraries obtained by splitting the source relational database; a second determining submodule 6052, configured to determine, based on a data amount of a data table in the source relational database, the data table used for storing the user data, a suggested number of sub tables obtained by splitting the data table; a third determining submodule 6053 configured to determine a suggested split key for the data table based on the access characteristic data for the user data in the source relational database over a historical period of time.

Optionally, the third determining sub-module 6053 is specifically configured to: determining a suggested split key for the data table based on a number of occurrences of a same field name in a same type of SQL statement that is updated for user data stored in the data table within the historical period of time.

Optionally, the target library includes a target logical library and a target physical library having a mapping relationship with the target logical library, and the establishing module 606 is specifically configured to: establishing the target logical library and the target physical library for storing the user data in the target distributed relational database based on a suggested number of libraries obtained by splitting the source relational database, a suggested number of tables obtained by splitting a data table in the source relational database, and a suggested split key for the data table.

Optionally, after the migrating module 607, the apparatus further includes: a second obtaining module 608, configured to obtain, from a log file of the source relational database, an SQL statement used for updating the user data in a migration time period of the user data, where the log file is used to record the SQL statement used for updating the user data; an executing module 609, configured to execute, for the target library in the target distributed relational database, an SQL statement used for updating the user data in the migration time period of the user data.

Referring to fig. 7, a schematic structural diagram of a data migration apparatus in a seventh embodiment of the present application is shown.

The data migration apparatus provided in this embodiment includes: a third receiving module 703, configured to receive an operation for instructing to migrate user data in the source relational database to the target distributed relational database; a sending module 704, configured to send, based on the operation, a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

Optionally, the third receiving module 703 includes: a first receiving sub-module 7031, configured to receive an operation on a data migration control shown in a page of a client, where the operation is used to instruct to migrate user data in the source relational database to the target distributed relational database.

Optionally, before the first receiving sub-module 7031, the apparatus further includes: a second receiving submodule 701, configured to receive a selection operation for a migration destination of the user data shown in a page of the client; and a refreshing sub-module 702, configured to refresh a page of the client based on the selection operation, so that the page of the client shows the data migration control.

Optionally, after the sending module 704, the apparatus further includes: a fourth receiving module 705, configured to receive a data migration response carrying a data migration completion message, which is returned by the data migration tool based on the data migration request; a switching module 706, configured to switch the source relational database to the target distributed relational database based on the data migration completion message carried in the data migration response.

Fig. 8 is a schematic structural diagram of an electronic device in an eighth embodiment of the present application; the electronic device may include:

one or more processors 801;

a computer-readable medium 802, which may be configured to store one or more programs,

when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data migration method according to the first embodiment, the second embodiment, or the third embodiment.

FIG. 9 is a diagram illustrating a hardware configuration of an electronic device according to a ninth embodiment of the present application; as shown in fig. 9, the hardware structure of the electronic device may include: a processor 901, a communication interface 902, a computer-readable medium 903, and a communication bus 904;

wherein the processor 901, the communication interface 902, and the computer readable medium 903 are in communication with each other via a communication bus 904;

alternatively, the communication interface 902 may be an interface of a communication module, such as an interface of a GSM module;

the processor 901 may be specifically configured to: determining splitting suggestion data of user data to be migrated from a source relational database to a target distributed relational database based on feature data of the user data to be migrated in the source relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database. Further, the processor 901 may be further configured to: receiving an operation for instructing migration of user data in a source relational database to a target distributed relational database; and sending a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

The Processor 901 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The computer-readable medium 903 may be, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code configured to perform the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code configured to carry out operations for the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may operate over any of a variety of networks: including a Local Area Network (LAN) or a Wide Area Network (WAN) -to the user's computer, or alternatively, to an external computer (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions configured to implement the specified logical function(s). In the above embodiments, specific precedence relationships are provided, but these precedence relationships are only exemplary, and in particular implementations, the steps may be fewer, more, or the execution order may be modified. That is, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a determination module, a setup module, and a migration module. The name of these modules does not constitute a limitation to the module itself in some cases, for example, the determining module may also be described as a module for determining splitting proposal data for migrating the user data from the source relational database to the target distributed relational database based on the characteristic data of the user data to be migrated in the source relational database.

As another aspect, the present application further provides a computer-readable medium, on which a computer program is stored, which when executed by a processor implements the data migration method as described in the first, second or third embodiment.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: determining splitting suggestion data of user data to be migrated from a source relational database to a target distributed relational database based on feature data of the user data to be migrated in the source relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database. Further, the apparatus is caused to: receiving an operation for instructing migration of user data in a source relational database to a target distributed relational database; and sending a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

The expressions "first", "second", "said first" or "said second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance, but these expressions do not limit the respective components. The above description is only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure.

When an element (e.g., a first element) is referred to as being "operably or communicatively coupled" or "connected" (operably or communicatively) to "another element (e.g., a second element) or" connected "to another element (e.g., a second element), it is understood that the element is directly connected to the other element or the element is indirectly connected to the other element via yet another element (e.g., a third element). In contrast, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is interposed therebetween.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A method of data migration, the method comprising:

determining splitting suggestion data of user data to be migrated from a source relational database to a target distributed relational database based on feature data of the user data to be migrated in the source relational database;

establishing a target library for storing the user data in the target distributed relational database based on the split proposal data;

migrating the user data from the source relational database to the target repository in the target distributed relational database.

2. The method of claim 1, wherein the characteristic data of the user data to be migrated comprises at least one of: the total amount of the user data in the source relational database, the amount of data in a data table for storing the user data in the source relational database, and the access characteristic data for the user data in the source relational database in a historical period.

3. The method of claim 2, wherein prior to the determining the split proposal data for the migration of the user data from the source relational database to the target distributed relational database, the method further comprises:

receiving an account and a password of the source relational database input by a user at a client;

and acquiring the total data amount of the user data and the data amount of the data table from the source relational database based on the account and the password of the source relational database.

4. The method of claim 2, wherein prior to the determining the split proposal data for the migration of the user data from the source relational database to the target distributed relational database, the method further comprises:

receiving access authority data aiming at the log file of the source relational database, which is input by a user at a client;

and extracting access characteristic data aiming at the user data in the historical time period from the log file based on the access authority data, wherein the log file is used for recording SQL statements for updating the user data in the source relational database.

5. The method of claim 4, wherein the access characteristic data for the user data over the historical time period comprises at least one of: the number of the SQL statements updated for the user data in the historical time period, the types of the SQL statements updated for the user data in the historical time period, and the number of times of occurrence of the same field name in the SQL statements updated for the user data in the historical time period.

6. The method of claim 1, wherein the split recommendation data comprises at least one of: the suggested number of sub-libraries obtained by splitting the source relational database, the suggested number of sub-tables obtained by splitting the data tables in the source relational database, and the suggested split key for the data tables.

7. The method of claim 1, wherein the determining split proposal data for migration of user data from a source relational database to a target distributed relational database based on characteristic data of the user data to be migrated in the source relational database comprises:

determining a suggested number of sub-databases obtained by splitting the source relational database based on the total amount of the user data in the source relational database and the potentialization data of the user data at a future time;

determining a suggested number of sub-tables obtained by splitting the data table based on a data amount of the data table for storing the user data in the source relational database;

determining a suggested split key for the data table based on access characteristic data for the user data in the source relational database over a historical period of time.

8. The method of claim 7, wherein the determining a suggested split key for the data table based on access characteristic data for the user data in the source relational database over a historical period of time comprises:

determining a suggested split key for the data table based on a number of occurrences of a same field name in a same type of SQL statement that is updated for user data stored in the data table within the historical period of time.

9. The method of claim 1, wherein the target repository includes a target logical repository and a target physical repository having a mapping relationship with the target logical repository,

the establishing a target library in the target distributed relational database for storing the user data based on the split proposal data comprises:

establishing the target logical library and the target physical library for storing the user data in the target distributed relational database based on a suggested number of libraries obtained by splitting the source relational database, a suggested number of tables obtained by splitting a data table in the source relational database, and a suggested split key for the data table.

10. The method of claim 1, wherein after the migrating the user data from the source relational database to the target one of the target distributed relational databases, the method further comprises:

acquiring SQL statements used for updating the user data in the migration time period of the user data from a log file of the source relational database, wherein the log file is used for recording the SQL statements used for updating the user data;

and executing SQL sentences used for updating the user data in the migration time period of the user data aiming at the target library in the target distributed relational database.

11. A method of data migration, the method comprising:

receiving an operation for instructing migration of user data in a source relational database to a target distributed relational database;

and sending a data migration request for requesting to migrate the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation, so that the data migration tool migrates the user data in the source relational database to the target distributed relational database based on the data migration request.

12. The method of claim 11, wherein the receiving an operation to indicate migration of user data in a source relational database to a target distributed relational database comprises:

receiving an operation for a data migration control shown in a page of a client, where the operation is used to instruct to migrate user data in the source relational database to the target distributed relational database.

13. The method of claim 12, wherein prior to the receiving operation for a data migration control exposed in a page of a client, the method further comprises:

receiving a selection operation for a migration destination of the user data shown in a page of the client;

and refreshing the page of the client based on the selection operation so as to enable the page of the client to display the data migration control.

14. The method of claim 11, wherein, after sending a data migration request to a data migration tool for requesting migration of user data in the source relational database to the target distributed relational database based on the operation, the method further comprises:

receiving a data migration response which is returned by the data migration tool based on the data migration request and carries a data migration completion message;

and switching the source relational database to the target distributed relational database based on the data migration completion message carried in the data migration response.

15. A data migration system, the system comprising:

a client and a data migration tool communicatively coupled to the client,

the client is used for receiving an operation for indicating the migration of user data in a source relational database to a target distributed relational database, and sending a data migration request for requesting the migration of the user data in the source relational database to the target distributed relational database to a data migration tool based on the operation;

and the data migration tool is used for receiving the data migration request sent by the client and migrating the user data in the source relational database to the target distributed relational database based on the data migration request.

16. The system according to claim 15, wherein the data migration tool is specifically configured to determine, based on feature data of user data to be migrated in a source relational database, split recommendation data for migrating the user data from the source relational database to a target distributed relational database; establishing a target library for storing the user data in the target distributed relational database based on the split proposal data; migrating the user data from the source relational database to the target repository in the target distributed relational database.

17. A data migration apparatus, the apparatus comprising:

the determining module is used for determining splitting suggestion data of the user data transferred from the source relational database to the target distributed relational database based on the characteristic data of the user data to be transferred in the source relational database;

the establishing module is used for establishing a target database for storing the user data in the target distributed relational database based on the splitting suggestion data;

a migration module to migrate the user data from the source relational database to the target library in the target distributed relational database.

18. An electronic device, the device comprising:

one or more processors;

a computer readable medium configured to store one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a data migration method as claimed in any one of claims 1-10, or to implement a data migration method as claimed in any one of claims 11-14.

19. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the data migration method according to any one of claims 1 to 10 or carries out the data migration method according to any one of claims 11 to 14.