WO2012083679A1

WO2012083679A1 - Data migration method, data migration device and data migration system

Info

Publication number: WO2012083679A1
Application number: PCT/CN2011/077102
Authority: WO
Inventors: 李丽娟
Original assignee: 华为技术有限公司
Priority date: 2011-07-13
Filing date: 2011-07-13
Publication date: 2012-06-28
Also published as: CN102308297B; CN102308297A

Abstract

Disclosed in the present invention are a data migration method, a data migration device and a data migration system. The method comprises: obtaining relational database table SCHEMA information from a relational database (101); mapping the SCHEMA information to a non-relational database XML model, and recording the corresponding mapping relation in a mapping relation table (102); loading the data in the relational database into the non-relational database XML model according to the mapping relation table. The technical problem of smoothly migrating the data in the relational database into the non-relational database is resolved, and the smooth conversion of service data is realized.

Description

Data migration method, data migration device and data migration system

The present invention relates to the field of data processing technologies, and in particular, to a data migration method, a data migration device, and a data migration system.

Background technique

As the scope of database applications expands, the functionality of relational databases is expanded, but the complexity and cost of use are also greatly increased. Many applications do not require the complex functions of a relational database, and are not limited by the data model of the relational database. Especially with the development of web2.0 services and the proliferation of network content, new demands have arisen on database systems, which require massive storage and concurrency, flexible expansion, high reliability, high availability, low cost, and flexible tables. Features such as structure, however, traditional relational databases cannot implement these features. Based on this, the industry has proposed various no sql non-relational databases, such as self-organizing, self-management, low cost, flexible scalability, mass storage, high concurrent access, and query relationship. Due to cost, scalability, efficiency, and customization of autonomous functions, many businesses currently want to switch data in a relational database to a non-relational database as much as possible. However, due to differences in data models, data in relational databases cannot be migrated directly to non-relational databases.

In the course of research and practice of the prior art, the inventors of the present invention have found that there is a need in the industry for a mapping of a data model between a relational database and a non-relational database.

Summary of the invention

In view of this, the embodiment of the present invention provides a data migration method, device, and data migration system, which can smoothly migrate data in a relational database to a non-relational database.

An embodiment of the present invention provides a data migration method, where the method includes:

Obtain the relational database table SCHEMA information from the relational database;

Mapping the SCHEMA information into a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table;

The data of the relational database is imported into the non-relational database according to the mapping relationship recorded in the mapping relationship table.

The embodiment of the invention further provides a data migration device, the device comprising:

a first obtaining unit, configured to obtain a relational database library table SCHEMA information from a relational database; a mapping unit, configured to map the SCHEMA information into a non-relational database XML model, and record a corresponding mapping relationship in the mapping relationship table;

The data migration unit is configured to import the data of the relational database into the non-relational database according to the mapping relationship recorded in the mapping relationship table.

The embodiment of the present invention further provides a data migration system, including at least one relational database and one non-relational database, and further comprising: a data migration device, configured to obtain a relational database library table SCHEMA information from a relational database; and map the SCHEMA information The non-relational database XML model is formed, and the corresponding mapping relationship is recorded in the mapping relationship table; according to the mapping relationship recorded in the mapping relationship table, the data of the relational database is imported into the non-relational database.

It can be seen from the above that the embodiment of the present invention provides a method for data migration between a relational database and a non-relational database, a data migration device, and a data migration system, which can be used to migrate data of a traditional relational database to a non-relational database. Smooth migration of data services. The data migration in the embodiment of the present invention is automatic, not only ensures the reliability of migration, but also improves the efficiency of data migration, thereby achieving smooth switching of services.

DRAWINGS

1 is a flowchart of a data migration method according to Embodiment 1 of the present invention;

2 is a flowchart of a method for data migration according to Embodiment 2 of the present invention;

FIG. 3 is a flowchart of a SCHEMA mapping in various modes according to an embodiment of the present invention; FIG. 4 is a flowchart of a data importing process according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a data migration system according to an embodiment of the present invention; FIG.

FIG. 7 is a structural diagram of an application example of providing a data migration system according to an embodiment of the present invention.

detailed description

The embodiments of the present invention are further described in detail below with reference to the accompanying drawings and embodiments.

Referring to FIG. 1 , a flowchart of a data migration method according to Embodiment 1 of the present invention is provided. The method includes:

Step 101: Obtain a relational database table SCHEMA information from a relational database; - an acquisition process in which the SCHEMA mapping engine first sends a request for obtaining SCHEMA information to the relational database; receiving the SCHEMA information fed back by the relational database.

Step 102: Mapping the SCHEMA information into a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table;

That is, the SCHEMA mapping engine maps the SCHEMA information to a relational database

The XML model; according to the definition of SCHEMA, the relational database XML model is mapped to the non-relational database XML model, and the corresponding mapping relationship is recorded in the mapping relationship table.

The mapping the relational database XML model to the non-relational database XML model includes: configuring the relational database XML model to be a normalized processing mode or a non-normalized processing mode, where

When configured as a normalization processing mode, each table in the relational database is mapped to a column family in the non-relational database, and the corresponding mapping relationship is recorded in the mapping relationship table;

When configured as a denormalized processing mode, maps tables in multiple relational databases to a column family in a non-relational database.

Step 103: Import the data of the relational database into the non-relational database XML model according to the mapping relationship table.

Preferably, before the step 101 of the foregoing embodiment, the method may further include: adding a definition of the non-relational data object table SCHEMA in the non-relational database; specifically: adding a SCHEMA constraint definition of a primary key of the column family; Add a SCHEMA constraint definition for a specific column of the column family; map the relational database table of one or more union primary keys to a super column family in a non-relational database.

Preferably, before the step 101 of the foregoing embodiment, the method may further include: establishing a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database, and recording the foregoing SCHEMA definition in the mapping relationship table. Mapping relations.

In step 103, the data of the relational database is imported into the non-relational database according to the mapping relationship recorded in the mapping relationship table, and specifically includes:

Generating a data definition command according to the non-relational database XML model; sending to a non-relational database - - sending the data definition command to create SCHEMA in a non-relational database; obtaining a data set from the relational database: specifically: scanning a non-relational database XML model and a mapping relation table, generating a SQL query statement, using the SQL The query statement obtains the data set from the relational database; and imports the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table. Preferably, after the step 103, the method may further include: optimizing the mapping relationship in the record in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode.

The embodiment of the invention provides a method for data migration between a relational database and a non-relational database, which can be used to migrate data of a traditional relational database to a non-relational database (such as an emerging Bigtable data model), thereby realizing data services. Smooth migration. The data migration in the embodiment of the present invention is automatic, not only ensures the reliability of migration, but also improves the efficiency of data migration, thereby realizing smooth switching of services.

Referring to FIG. 2, a flowchart of a data migration method according to Embodiment 2 of the present invention is provided. The method includes:

Step 201: Add a SCHEMA definition to the non-relational database;

In this embodiment, the non-relational database takes the class Bigtable data as an example, but is not limited thereto. That is to say, the SCHEMA mapping engine first adds the SCHEMA definition to the non-relational database of the Bigtable data model, and the specific process is:

Since the current non-relational database of the Bigtable data model can only define Keyspace (key-value space) and column family (歹 ll family), these two definitions can conceptually correspond to db (database) and table Table in relational database. In order to realize the mapping of relational tables in the database to non-relational tables in non-relational databases, it is necessary to add the SCHEMA constraint definition of non-relational database tables in non-relational databases. The specific definition process is:

First, increase the SCHEMA constraint definition for the column family's primary key (row key), corresponding to the primary key definition of the relational database;

Then, increase the SHCEMA constraint definition for the specific column under the column family, that is, define a - - column column specific column and its column name, type, length, whether it can be empty, default value, etc., corresponding to the relational database column SCHEMA definition;

Finally, in order to map a relational database table of one or more federated primary keys to a super column family in a non-relational database, it is necessary to increase the constraint definition of the relational database table to be converted into a super column family. Constraint. In the SCHMEMA definition of a relational database table, when there is only one union primary key, the value of the union primary key is mapped to the name of the super column; when there are more than two union primary key definitions, the value of the union primary key is cascaded and mapped to super The name of the column, the order of the cascading can be specified, the higher the cascading field, the faster the query will search for it.

Step 201: Establish a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database, and record the mapping relationship defined by the SCHEMA in the mapping relationship table;

The mapping relationship between the relational table and the non-relational table established by the SCHEMA mapping engine is as shown in Table 1 to Table 3:

Table 1 :

User basic information table i space— user— info

Ispace_user_info

Table 2:

Unified message table ispace message info

Field Type Length Description Information

# userid Varchar2 20

# m sg id Varchar2 40 - -

Ispace_message_info

Userid

table 3:

Unified Messaging Status Report Table i space— message— recv n o o 〇

Ispace晷_message_recv

Userid

Userid recvl/001 recv2/001 recvl/002 rec 001 (superCol) (superCol) (superCol) (su

State state state state val val val val userid

002 Step 203: Obtain SCHEMA information from a relational database, and map the SCHEMA information into an XML model.

That is, when starting the SCHEMA mapping, the SCHEMA mapping engine first requests the SCHEMA information from the relational database and maps it to the XML model.

Among them, there are more than 4 methods for mapping relational database tables to standard XML models. For example, .net, Delphi, etc. all provide automatic conversion methods, basically one-to-one mapping. Of course, you can also refer to their implementation or direct call. The method provided by the embodiment of the present invention is not limited. In this embodiment, the mapping conversion rule is: The database corresponds to the XML db node, and the node contains description information of all Tables in the database; each relationship table corresponds to one of the XML A table node, the node contains the SCHEMA information of each row in the table; each row in the table corresponds to a column node of the XML, and the one-to-one mapping in the node describes the SCHEMA of each row in the corresponding relation table, wherein the column is the primary key , then the booster port isRowKey = true, :3⁄4 mouth J3⁄4I column is the joint primary key, and the shell ¹ J booster port isUnionKey = tme.

For example, the Unified Messaging Status Report Table is shown in Table 4.

Table 4

Message status report table i space— message— recv

Table 4 above can be implemented by the following procedure:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<column name = "userid" type = "varchar2" length = 20 nullable = false isRowKey = true /> - -

</table>

</db>

Step 204: Mapping the XML model into a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table;

This embodiment provides a configuration interface, which can configure the default mapping mode of the SCHEMA mapping engine to be a normalized processing mode or a non-normalized processing mode; and can also implement a customizable mapping process and align the mapping results.

1) Configure the default mapping mode of the SCHEMA mapping engine to be normalized processing mode

First, when the SCHEMA mapping engine is configured to normalize processing mode, each table in the relational database is mapped to a column family of non-relational databases.

According to the SCHEMA definition of the non-relational database, the SCHEMA mapping engine sequentially scans the XML files generated by direct mapping, and maps each database db node scanned to a non-relational database Keyspace node with default attribute definitions, and these attributes contain at least the number of copies. , copy distribution strategy; map each table node scanned in a db into a column family node, the column family column type should be determined according to the column SCHEMA definition in the table, if the column definition contains the union field (join primary key) ), the column type is "super", otherwise it is "standard". When a table node is mapped into a column family node, the necessary mode attributes must be added. These attributes include at least the column sort type, cache configuration, etc. Each column node scanned in a table node is directly mapped into a column node. If a column node is a union primary key, the non-relational database XML is described as isKey = true, and the first scanned joint primary key is added to the isPrefix. = true , other union primary keys isPrefix = false. - - The above process can be implemented by the following procedures, but is not limited thereto.

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

</table>

</db>

The following procedure can be obtained by the above procedure:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

< keyspace name = "messagedb"

Replication—factor = 3

Strategy_class="org.apache.cassandra.locator.SimpleStrategy"> <columnfamily name = "ispace_msg_recvState"

ColumnType = "Super"

Compare With = "BytesType"

CompareSubcolumnsWith = "BytesType"

KeysCached="50%"

RowsCached=" 10000" >

<column name ="userid" isRowKey = "true" type = "string" length = 20 nullable="false"/> - -

< /columnfamily>

< /keyspaco

In the above program segment, the first segment xml program is a relation table xml file directly mapped into a relational database table, and the second segment is a non-relational table xml file to which a default attribute value is mapped; mapping into a non-relational table xml file Then you can directly map to non-relational Keyspace and columnfamily, and create a non-relational table accordingly.

Secondly, a new mapping relationship table is created before starting the mapping process for each database, and the mapping relationship between the relational table and the non-relational table in each db is recorded. Each time the SCHEMA mapping engine completes a table mapping, the mapping relationship is recorded in the mapping table in the data migration device. As shown in Table 5, Table 5

2) Configure the default mapping mode of the SCHEMA mapping engine to be denormalized processing mode

In this embodiment, the relational data model design should follow the principle of minimizing data redundancy, that is, normalization. The public data is all extracted and stored in a separate table, and other tables are associated with it. Non-relational databases use distributed servers to form distributed clusters, have flexible scalability, allow data redundancy, and in order to improve search efficiency, data that can be stored in a table is generally stored in a table, so the relationship is In the process of mapping database tables to non-relational database tables, - - The operational efficiency of data after high migration, to meet the principle of non-relational database modeling, it is necessary to map multiple relational database tables into a non-relational database column family, that is, denormalized or redundant processing.

That is to say, when the configuration mapping default mode is denormalized mode, when the SCHEMA mapping engine is executed, all foreign keys in the relational table should be removed, and the data in the common table should be distributed to the table where the foreign key is located.

Similarly, a mapping relationship table is created before starting the mapping process for each database, and the mapping relationship between the relational table and the non-relational table in each db is recorded. Each time the SCHEMA mapping engine completes a table mapping, the mapping relationship is recorded in the established mapping table.

According to the SCHEMA definition of the non-relational database, the SCHEMA mapping engine sequentially scans the XML files generated by direct mapping, and maps each database db node scanned to a non-relational database Keyspace node with default attribute definitions, and these attributes contain at least the number of copies. , copy distribution strategy; check whether there is a isForeignKey = true column in each table node scanned in a db, if there is, first fill in the foreign key in the "association table" column of the corresponding row in the mapping relationship table The referenced table, and the "no mapping" column in the row of the foreign key reference table is assigned to true, as shown in Table 6; if not, the one-to-one mapping is performed with reference to the normalization processing.

Table 6

In this embodiment, the column type of the column family needs to be determined according to the column SCHEMA definition in the table. If the column definition includes a union field (join primary key), the column type is "super", otherwise it is "standard". When a table node is mapped into a column family node, it is necessary to add the necessary mode attributes, which include at least the column sort type and the cache configuration; - - Each column scanned in the point maps directly to the column node. If a column node is a union primary key, the non-relational database XML is described as isKey = tme, and the first scanned joint primary key is added to the isPrefix. = true, other union primary keys isPrefix = false. If it is checked in the mapping relation table that the table being executed has an associated table, all the columns in the associated table are mapped as ordinary columns to the non-relational database XML file, and the original foreign key definition is canceled. This process can be improved on Program 1 and Program 2 to get Program 3, as follows:

Procedure 1

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

</table>

</db>

Procedure 2

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<column name = "size" type = "number" length = 4 nullable = false /><column name = "desc" type = "varchar2" length = 40 nullable = false - -

/>

</table>

</db>

Procedure 3

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<keyspace name = "messagedb"

Replication—factor = 3

ColumnType = "Super"

Compare With = "BytesType"

CompareSubcolumnsWith = "BytesType"

KeysCached="50%"

RowsCached=" 10000" >

<column name ="userid" isRowKey = "true" type = "string" length =

20 nullable=" false" />

< /keyspaco

Step 205: Generate a data definition command according to the non-relational database XML model. Step 206: Send the data definition command to a non-relational database, and create a non-relational database. - - SCHEMA;

Step 207: Obtain a data set from the relational database.

Step 208: Import the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.

Preferably, after the step 204, when the relational database XML model is configured as a normalized processing mode or a non-normalized processing mode, the mapping relationship in the record in the mapping relationship table is optimized:

That is to say, under the premise that the SCHEMA mapping engine performs the normalization processing mode/denormalization, the mapping relationship table can also be modified through the configuration interface, so that the specified part is denormalized/normalized under the condition that most of the execution of the normalized/non-normalized mapping is guaranteed. Mapping, the specific implementation process is:

1) The default mode of the configuration data migration device is SCHEMA mapping mode, the SCHEMA mapping engine completes the automatic SCHEMA mapping, generates the non-relational database XML file, and fills in the mapping relationship table;

2) Modify the mapping relationship table, specify the partial table to perform the denormalization (normalized processing mode) / normalization (in the non-normalized processing mode) processing mode, and then configure the data migration device to map and migrate the mode, the SCHEMA mapping engine starts the automatic mapping, Generating a non-relational database XML file according to the already filled mapping table;

3) The trigger executor performs data migration according to the mapping relationship.

The configuration interface provided by the data migration device can also modify the default attribute definitions of Keyspace and column family to be tuned according to the needs of the service. The specific execution process is as follows:

The configuration data migration device is in the SCHEMA mapping mode, the SCHEMA mapping engine automatically completes the data model mapping, generates the non-relational database XML file, and fills in the mapping relationship table;

Modify the attributes according to the business requirements, and then configure the data migration mode.

The trigger executor migrates data according to the mapping relationship.

In order to facilitate the understanding of those skilled in the art, the following specific examples are given.

Referring to FIG. 3, a flow chart of SCHEMA information mapping into a non-relational database XML in various modes is provided according to an embodiment of the present invention. In this embodiment, SCHEMA is mapped to various modes. - - The process of shooting is described in detail, as shown in the figure, including:

Step 301: Determine whether it is a normalization mode; if yes, go to step 302; otherwise, go to step 303;

Step 302: Set the SCHEMA mapping engine to the normalization mode;

Step 303: Set the SCHEMA mapping engine to a denormalized mode;

Step 304: The SCHEMA mapping engine determines the mapping mode. If it is the SCHEMA mapping mode, step 305 is performed: if it is the mapping and data migration mode, step 310 is performed;

Step 305: Obtain SCHEMA information from a relational database, and directly map the SCHEMA information into a relational data XML model;

Step 306: Map the relational data XML model to a non-relational data model XML according to a specification mode and a mapping relationship table configuration;

Step 307: Determine whether there is a corresponding item in the mapping relationship table, and if yes, go to step 308; otherwise, go to step 309;

Step 308: Perform mapping according to the configuration in the correspondence table, and end;

Step 309: Perform mapping according to the configuration in the specification mode, and fill in the mapping relationship correspondence table, and end; Step 310: Obtain SCHEMA information from the relational database, and directly map the SCHEMA information into a relational data XML model;

Step 311: Mapping the relational data XML model into a non-relational data model XML according to a specification mode and a mapping relationship table configuration;

Step 312: Determine whether there is a corresponding item in the mapping relationship table, and if yes, go to step 313; otherwise, go to step 314;

Step 313: Perform mapping according to the configuration in the correspondence table, and then perform step 315; Step 314: Perform mapping according to the configuration in the specification mode, and fill in the mapping relationship correspondence table, and then perform step 315;

Step 315: Trigger the executor to perform the data migration process.

That is, Figure 3 depicts the SCHEMA mapping process from a relational database to a non-relational database. a data import process from a relational database to a non-relational database of the Bigtable data model, - - In, the data import process is performed by an executor in the data migration device, including:

1) The executor issues a data definition command to the non-relational database, and creates a SCHEMA information in the non-relational database according to the non-relational database XML model generated by the SCHEMA mapping process described above;

2) The actuator sequential scan directly maps each table of the XML file, and looks at the mapping relationship table to determine whether the table does not need to be mapped. If not, ignore the table and continue scanning the next table; if this table is the last table , then end; if mapping is required, perform step 3);

3) judging whether the table has an associated table, if yes, generating an SQL query according to the directly mapped XML file, extracting the data set of the source table and its associated table from the relational database; if not, generating according to the directly mapped XML file SQL query, extracting the data set of the source table from the relational database; 4) generating the data import mode of the non-relational database according to the corresponding non-relational data model XML, actively calling the data import interface of the non-relational database according to the SCHEMA mapping relationship, and putting the data set Import a non-relational database;

5) Determine if there is any table that needs to be mapped. If there is, return to step 1, otherwise end. The specific data importing process is shown in FIG. 4. FIG. 4 is a flowchart of data import according to an embodiment of the present invention, which specifically includes:

Step 401: Trigger an executor to perform data migration;

Step 402: Create a non-relational data table by calling a non-relational database interface according to the non-relational data model XML.

Step 403: sequentially scan a table table directly mapping XML, and view the mapping relationship table, determine whether the table Table does not need to be mapped; if yes, execute step 410; otherwise, perform step 404; Step 404: determine whether the table Table is associated Table, if yes, go to step 405; otherwise, go to step 406;

Step 405: Generate SQL that extracts data from the source table and its associated table, and extracts the data set from the relational database;

Step 406: Generate a source table, extract data SQL, and extract a data set from the relational database. Step 407: Generate a data import function to the non-relational database according to the corresponding non-relational data model XML; - Step 408: Loop out the data strip from the data set, and use the import function to import the data strip; Step 409: Determine whether there is still a table needs to be scanned, and if so, return to step 403; No, go to 'J, go to step 411;

Step 410: Determine whether there is still a table to scan, and if yes, return to step 403; Step 411;

Step 411: End.

Based on the implementation process of the foregoing embodiment, a data migration device is also provided in the embodiment of the present invention. The structure of the data migration device is as shown in FIG. 5. The device includes: a first acquiring unit 51, a mapping unit 52, and a data migration unit 53, The first obtaining unit 51 is configured to obtain the relational database table SCHEMA information from the relational database. The mapping unit 52 is configured to map the SCHEMA information into a non-relational database XML model, and in the mapping relationship table. Recording the corresponding mapping relationship; the data migration unit 53 is configured to import the data of the relational database into the non-relational database according to the mapping relationship recorded in the mapping relationship table.

The unit of the above device may be implemented by a processor, a memory or a special purpose circuit or the like.

Preferably, the device may further include: a defining unit, connected to the first obtaining unit, configured to add a definition of the non-relational data object table SCHEMA in the non-relational database, specifically: adding a primary key to the column family SCHEMA constraint definition; adds SCHEMA constraint definitions for specific columns of the column family; maps one or more union primary key's relational database tables to super column families in non-relational databases. The specific definition process is as described above and will not be described here.

Preferably, the device may further include: an establishing unit, connected to the mapping unit, configured to establish a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database.

The first obtaining unit includes: a first sending unit and a first receiving unit, where the first sending unit is configured to send a request for acquiring SCHEMA information to the relational database; the first receiving unit And the SCHEMA information for receiving the feedback of the relational database.

The mapping unit includes: a first mapping unit and a second mapping unit, where the first mapping unit is configured to map the SCHEMA information into a relational database XML model; - a second mapping unit, configured to map the relational database XML model to a non-relational database XML model according to the definition of SCHEMA, and record the corresponding mapping relationship in the mapping relationship table.

The second mapping unit further includes: a normalization processing mode mapping unit and/or a non-normalization processing mode mapping unit, where the normalization processing mode mapping unit is configured to configure the relational database XML model into a normalized processing mode, and Mapping each table in the relational database to a column family in the non-relational database, and recording a corresponding mapping relationship in the mapping relationship table; the denormalized processing mode mapping unit is configured to configure the relational database XML model For the non-normalized processing mode, the tables in multiple relational databases are mapped to a column family in the non-relational database, and the corresponding mapping relationship is recorded in the mapping relation table.

Preferably, the device may further include: a configuration optimization unit, configured to be connected to the second mapping unit, configured to be in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode The mapping relationship in the record is optimized.

In the above embodiment, the first acquiring unit includes: a scanning unit, a generating unit, and a data acquiring unit, wherein the scanning unit is configured to scan a non-relational database XML model and a mapping relationship table to obtain a scan result; a generating unit, configured to generate an SQL query statement according to the scan result, where the data acquiring unit is configured to obtain a data set from the relational database by using the SQL query statement.

In the above embodiment, the data migration unit includes: a generating unit, a creating unit, a second obtaining unit, and a data importing unit, wherein the generating unit is configured to generate a data definition command according to the non-relational database XML model; The creating unit is configured to send the data definition command to a non-relational database to create a SCHEMA in a non-relational database; the second obtaining unit is configured to obtain a data set from the relational database; the data And an importing unit, configured to import the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.

For the implementation process of the functions and functions of the various units in the device, refer to the implementation process of the corresponding steps in the foregoing method, and details are not described herein again.

Correspondingly, an embodiment of the present invention further provides a data migration system, and a schematic structural diagram thereof is shown in FIG. 6 . - The system comprises: at least one relational database (in the figure, the relational databases 61 and 62 are exemplified, but not limited thereto), and at least one non-relational database (in the figure, the non-relational databases 631 and 632 are taken as an example, but It is not limited to this and the data migration device 63. The data migration device 63 is configured to obtain the relational database table SCHEMA information (such as the relational database 61) from the relational database; respectively map the SCHEMA information into a non-relational database XML model, and record correspondingly in the mapping relationship table. Mapping relationship; according to the mapping relationship table, data of a relational database (such as relational database 61) is imported into a non-relational database (such as non-relational database 631). When the data is not limited to this, the data of the non-relational database may be migrated to a plurality of non-relational databases, or the data in the plurality of relational databases may be migrated to a non-relational database, and the implementation process is similar. limit.

The data migration device in the system is the same as the data migration device in the foregoing embodiment, and is not described here.

The data migration apparatus in the embodiment of the present invention can implement the data migration requirement from the relational database to the non-relational database according to the modeling difference between the relational database and the non-relational database, and implements the multi-table from the relational database to the non-relational database. Table mapping mechanism. The traditional relational data model is designed to minimize data redundancy, that is, the principle of normalization. Public data is extracted and stored in a single table, and other tables are associated with it. Non-relational databases use inexpensive servers to form distributed clusters, allowing data redundancy, and in order to improve search efficiency, data that can be integrated into a table is generally not stored in a table, so the mapping process from relational database tables to non-relational database tables In order to improve the operational efficiency of the data after migration, it is necessary to map multiple relational database tables into a non-relational database column family, that is, the principle of denormalization. The apparatus of the embodiment of the present invention can automatically store the common table to the table where the foreign key is located. The operational efficiency of the data after migration and the convenient access mode are fully guaranteed.

FIG. 7 is a schematic structural diagram of a data migration system according to an embodiment of the present invention. The system includes: at least one data relation library (in this embodiment, a data relation library is taken as an example) 71. At least one non-data The relational library (this embodiment takes a non-data relational library as an example) 72, and the data migration means 73 respectively connected to the data relational library and the non-data relational library, but is not limited thereto. The above means may be implemented by a processor, a memory or a special purpose circuit or the like. - In this embodiment, the relational database 71: may be MySql, DB2, Oracle, etc., using a two-dimensional table to represent entities and their associations: using a table to represent a set of entities, a column to represent attributes, a table structure to represent a solid type ; Use specific redundancy information between tables to represent the connections between entities (primary keys, foreign keys). The advantages are: integrity is easy to express; single, the concept of the table is intuitive, single, user-friendly; solid theoretical basis (relational algebra); non-procedural data request, request data can not specify the path; high data independence , users only need to ask "what to do", no need to explain "how to do"; support collection operations.

The non-relational database 72: may be a non-relational database, a cloud database, or a no sql database whose data model is a class Bigtable. The non-relational database of the Bigtable data model is a cluster composed of inexpensive servers, and the cluster can self-organize and self-manage, and provide services as a whole. Such databases are characterized by low cost, flexible scalability, mass storage, high concurrency, autonomous feature customization, and weak affinity.

The data migration device 73: may be a hardware device or a software device, and is used to implement a data mapping and migration process between the relational database and the non-relational database of the Bigtable data model. The specific process is to obtain the SCHEMA information from the relational database, map it to the non-relational database SCHEMA, and extract the data from the relational database according to the mapping relationship to import into the non-relational database. The extracting the data from the relational database according to the mapping relationship into the non-relational database includes: generating a data definition command according to the non-relational database XML model; sending the data definition command to the non-relational database, creating SCHEMA in a non-relational database; obtaining a data set from the relational database; importing the data set into the SCHEMA of the non-relational database according to a corresponding mapping relationship recorded in the mapping relationship table.

The data migration device 73 includes: a first obtaining unit 731, configured to obtain a relational database table SCHEMA information from a relational database; the specific acquisition process is as described above, and details are not described herein; the SCHEMA mapping engine (ie, mapping) a unit 732, configured to map the SCHEMA information into a non-relational database XML model; an executor (ie, a data migration unit) 733, configured to import the data of the relational database into the non-relational database XML model according to the mapping relationship table in.

The data migration device 73 may further include a mapping relationship table 730, which is established by the establishing unit; the mapping relationship table records mapping the SCHEMA information into a non-relational database XML - - The mapping relationship of the model.

The data migration device 73 may further include a definition unit for adding a non-relational data object table, a definition of SCHEMA, and the like in the non-relational database.

The device may further include: a configuration optimization unit, configured to optimize the mapping relationship in the record in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode, where the configuration may be specifically configured Interface 734.

The embodiment of the invention provides a method, a device and a system for data migration between a relational database and a non-relational database, which can be used for data migration between a traditional relational database and a non-relational database of a Bigtable data model. Through the data migration device in the embodiment of the present invention, the service can perform data migration in a reliable, fast, and convenient manner, smoothly transition to a non-relational database, and utilize a non-relational database for low cost, flexible scalability, and high High efficiency and other advantages enhance the competitiveness of the business.

In the embodiment of the present invention, the business data in the database can be switched to the non-relational database of the Bigtable data model at any time, and the entire data migration process is automatic, which also saves cost and reduces error rate, and realizes Smooth migration of business data.

Further, in the embodiment of the present invention, the data migration device further provides a configuration interface, which can customize the data migration mode, and optimize the data table according to the service requirement, and also modify the mapping relationship table through the configuration interface, thereby adding more Attributes that implement more custom mappings. This configurable data migration process enables the business to migrate data from a relational database to a non-relational database, and then migrate data according to the modeling characteristics of the non-relational database, giving full play to the advantages of the non-relational database and improving the migrated data. Access efficiency.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by hardware, or can be implemented by means of software plus necessary general hardware platform. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (for example, may be a read only memory (ROM), a USB flash drive, a mobile hard disk, a random access memory (RAM), a disk or an optical disk, etc., which can store program code, etc., including a plurality of instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute the present The methods described in the various embodiments are invented. In the several embodiments provided herein, it is to be understood that the disclosed systems, devices, and methods may be implemented in other ways without departing from the spirit and scope of the application. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be another division manner, for example, multiple modules or components may be combined or Can be integrated into another system, or some features can be ignored, or not executed. The modules described as separate components may or may not be physically separated. The components displayed as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. . Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, the described systems, apparatus, and methods, and the schematic diagrams of various embodiments, may be combined or integrated with other systems, modules, techniques or methods without departing from the scope of the present application. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or otherwise.

It will be apparent to those skilled in the art that the various units or steps of the present invention described above can be implemented by a general-purpose computing device that can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they may be implemented by program code executable by the computing device, such that they may be stored in the storage device by the computing device, or they may be separately fabricated into individual integrated circuit modules, or they may be Multiple units or steps are made in a single integrated circuit module. Thus, the invention is not limited to any specific combination of hardware and software. The above is only the preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims

Rights request

A method for data migration, comprising:

2. The method of claim 1, further comprising: adding a definition of the non-relational database table SCHEMA to the non-relational database.

3. The method according to claim 2, wherein the adding the non-relational database table SCHEMA in the non-relational database includes:

Add a SCHEMA constraint definition for the primary key of the column family column family;

Add a SCHEMA constraint definition for the specific column field of the column family; map the relational database table with one or more union primary keys to the super 歹 'J family super column family in the non-relational database.

4. The method according to claim 3, further comprising:

Establish a mapping relationship table between the relational table in the relational database and the non-relational relational database of the non-relational database, and record the mapping relationship defined by the SCHEMA in the mapping relational table.

5. The method according to claim 1, wherein the obtaining the relational database table SCHEMA information from the relational database comprises:

Sending a request for obtaining SCHEMA information to the relational database;

Receiving the SCHEMA information fed back by the relational database.

The method according to any one of claims 2 to 5, wherein the mapping the relational database table SCHEMA information to a non-relational database XML model comprises:

Mapping the SCHEMA information into a relational database XML model;

Mapping the relational database XML model to a non-relational relationship according to the definition of SCHEMA The database XML model, and record the corresponding mapping relationship in the mapping relationship table.

The method according to claim 6, wherein the mapping the relational database XML model to a non-relational database XML model comprises: configuring the relational database XML model to a normalized processing mode or a non-normalized processing mode , among them,

8. The method according to claim 7, further comprising

When the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode, the mapping relationship in the records in the mapping relationship table is optimally configured.

The method according to any one of claims 2 to 8, wherein the importing the data of the relational database into the non-relational database according to the mapping relationship recorded in the mapping relationship table comprises:

Generating a data definition command according to the non-relational database XML model;

Sending the data definition command to a non-relational database to create a SCHEMA in a non-relational database;

Obtaining a data set from the relational database;

The data set is imported into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.

The method according to any one of claims 1 to 9, wherein the obtaining the data set from the relationship database comprises:

Scanning the non-relational database XML model and the mapping relation table, generating a SQL query statement, and obtaining the data set from the relational database by using the SQL query statement.

11. A data migration device, comprising:

a first obtaining unit, configured to obtain a relational database library table SCHEMA information from a relational database; a mapping unit, configured to map the SCHEMA information into a non-relational database XML model, And recording the corresponding mapping relationship in the mapping relationship table;

The device according to claim 11, wherein the device further comprises: a definition unit, configured to add a definition of the non-relational database table SCHEMA in the non-relational database, specifically comprising: adding a column family The SCHEMA constraint definition for the primary key; the SCHEMA constraint definition for the specific column of the column family; the relational database table for one or more union primary keys is mapped to the super column family in the non-relational database.

The device according to claim 11 or 12, wherein the device further comprises: an establishing unit, connected to the mapping unit, configured to establish a non-relational table between the relational table and the non-relational database in the relational database Mapping table.

The device according to claim 11 or 12, wherein the first acquiring unit comprises:

a first sending unit, configured to send a request for acquiring SCHEMA information to the relational database; and a first receiving unit, configured to receive the SCHEMA information fed back by the relational database.

The apparatus according to claim 12, wherein the mapping unit comprises: a first mapping unit, configured to map the SCHEMA information into a relational database XML model; and a second mapping unit, configured to follow SCHEMA Defining, mapping the relational database XML model to a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table.

The device according to claim 12, wherein the second mapping unit comprises: a normalization processing mode mapping unit, configured to configure the relational database XML model into a normalized processing mode, and Each table is mapped to a column family in a non-relational database, and the corresponding mapping relationship is recorded in the mapping relationship table; and/or

a non-normalized processing mode mapping unit, configured to configure the relational database XML model to be a non-normalized processing mode, and map the tables in the multiple relational databases to a column family in the non-relational database, and record the corresponding in the mapping relationship table Mapping relationship.

The device according to claim 16, wherein the device further comprises: a configuration optimization unit, configured to configure the relational database XML model as a normalized processing mode Or, in the non-normalized processing mode, optimize the mapping relationship in the records in the mapping table.

The device according to any one of claims 11 to 17, wherein the first obtaining unit comprises:

a scanning unit, configured to scan a non-relational database XML model and a mapping relationship table, to obtain a scan result;

a generating unit, configured to generate a SQL query statement according to the scan result;

a data obtaining unit, configured to acquire a data set from the relational database by using the SQL query statement.

The apparatus according to any one of claims 11 to 17, wherein the data migration unit comprises:

a generating unit, configured to generate a data definition command according to the non-relational database XML model; a creating unit, configured to send the data definition command to the non-relational database, and create a SCHEMA in the non-relational database;

a second obtaining unit, configured to second acquire a data set from the relational database;

And a data importing unit, configured to import the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.

20. A data migration system, comprising at least one relational database and one non-relational database, further comprising: a data migration device, configured to obtain a relational database library table SCHEMA information from a relational database; and map the SCHEMA information The non-relational database XML model is formed, and the corresponding mapping relationship is recorded in the mapping relationship table; according to the mapping relationship recorded in the mapping relationship table, the data of the relational database is imported into the non-relational database.

The system according to claim 20, wherein the data migration device is the data migration device according to any one of claims 11 to 19.