WO2012083679A1 - Procédé, dispositif et système de migration de données - Google Patents

Procédé, dispositif et système de migration de données Download PDF

Info

Publication number
WO2012083679A1
WO2012083679A1 PCT/CN2011/077102 CN2011077102W WO2012083679A1 WO 2012083679 A1 WO2012083679 A1 WO 2012083679A1 CN 2011077102 W CN2011077102 W CN 2011077102W WO 2012083679 A1 WO2012083679 A1 WO 2012083679A1
Authority
WO
WIPO (PCT)
Prior art keywords
relational database
mapping
mapping relationship
data
schema
Prior art date
Application number
PCT/CN2011/077102
Other languages
English (en)
Chinese (zh)
Inventor
李丽娟
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN2011800012395A priority Critical patent/CN102308297B/zh
Priority to PCT/CN2011/077102 priority patent/WO2012083679A1/fr
Publication of WO2012083679A1 publication Critical patent/WO2012083679A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/80—Information retrieval; Database structures therefor; File system structures therefor of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML
    • G06F16/84—Mapping; Conversion
    • G06F16/86—Mapping to a database

Definitions

  • the present invention relates to the field of data processing technologies, and in particular, to a data migration method, a data migration device, and a data migration system.
  • relational databases As the scope of database applications expands, the functionality of relational databases is expanded, but the complexity and cost of use are also greatly increased. Many applications do not require the complex functions of a relational database, and are not limited by the data model of the relational database. Especially with the development of web2.0 services and the proliferation of network content, new demands have arisen on database systems, which require massive storage and concurrency, flexible expansion, high reliability, high availability, low cost, and flexible tables. Features such as structure, however, traditional relational databases cannot implement these features. Based on this, the industry has proposed various no sql non-relational databases, such as self-organizing, self-management, low cost, flexible scalability, mass storage, high concurrent access, and query relationship.
  • the embodiment of the present invention provides a data migration method, device, and data migration system, which can smoothly migrate data in a relational database to a non-relational database.
  • An embodiment of the present invention provides a data migration method, where the method includes:
  • the data of the relational database is imported into the non-relational database according to the mapping relationship recorded in the mapping relationship table.
  • the embodiment of the invention further provides a data migration device, the device comprising:
  • a first obtaining unit configured to obtain a relational database library table SCHEMA information from a relational database
  • a mapping unit configured to map the SCHEMA information into a non-relational database XML model, and record a corresponding mapping relationship in the mapping relationship table
  • the data migration unit is configured to import the data of the relational database into the non-relational database according to the mapping relationship recorded in the mapping relationship table.
  • the embodiment of the present invention further provides a data migration system, including at least one relational database and one non-relational database, and further comprising: a data migration device, configured to obtain a relational database library table SCHEMA information from a relational database; and map the SCHEMA information
  • a data migration device configured to obtain a relational database library table SCHEMA information from a relational database; and map the SCHEMA information
  • the non-relational database XML model is formed, and the corresponding mapping relationship is recorded in the mapping relationship table; according to the mapping relationship recorded in the mapping relationship table, the data of the relational database is imported into the non-relational database.
  • the embodiment of the present invention provides a method for data migration between a relational database and a non-relational database, a data migration device, and a data migration system, which can be used to migrate data of a traditional relational database to a non-relational database. Smooth migration of data services.
  • the data migration in the embodiment of the present invention is automatic, not only ensures the reliability of migration, but also improves the efficiency of data migration, thereby achieving smooth switching of services.
  • FIG. 1 is a flowchart of a data migration method according to Embodiment 1 of the present invention.
  • FIG. 2 is a flowchart of a method for data migration according to Embodiment 2 of the present invention.
  • FIG. 3 is a flowchart of a SCHEMA mapping in various modes according to an embodiment of the present invention
  • FIG. 4 is a flowchart of a data importing process according to an embodiment of the present invention
  • FIG. 5 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a data migration system according to an embodiment of the present invention.
  • FIG. 7 is a structural diagram of an application example of providing a data migration system according to an embodiment of the present invention.
  • FIG. 1 a flowchart of a data migration method according to Embodiment 1 of the present invention is provided.
  • the method includes:
  • Step 101 Obtain a relational database table SCHEMA information from a relational database; - an acquisition process in which the SCHEMA mapping engine first sends a request for obtaining SCHEMA information to the relational database; receiving the SCHEMA information fed back by the relational database.
  • Step 102 Mapping the SCHEMA information into a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table;
  • the SCHEMA mapping engine maps the SCHEMA information to a relational database
  • the XML model according to the definition of SCHEMA, the relational database XML model is mapped to the non-relational database XML model, and the corresponding mapping relationship is recorded in the mapping relationship table.
  • the mapping the relational database XML model to the non-relational database XML model includes: configuring the relational database XML model to be a normalized processing mode or a non-normalized processing mode, where
  • each table in the relational database is mapped to a column family in the non-relational database, and the corresponding mapping relationship is recorded in the mapping relationship table;
  • Step 103 Import the data of the relational database into the non-relational database XML model according to the mapping relationship table.
  • the method may further include: adding a definition of the non-relational data object table SCHEMA in the non-relational database; specifically: adding a SCHEMA constraint definition of a primary key of the column family; Add a SCHEMA constraint definition for a specific column of the column family; map the relational database table of one or more union primary keys to a super column family in a non-relational database.
  • the method may further include: establishing a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database, and recording the foregoing SCHEMA definition in the mapping relationship table. Mapping relations.
  • step 103 the data of the relational database is imported into the non-relational database according to the mapping relationship recorded in the mapping relationship table, and specifically includes:
  • the method may further include: optimizing the mapping relationship in the record in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode.
  • the embodiment of the invention provides a method for data migration between a relational database and a non-relational database, which can be used to migrate data of a traditional relational database to a non-relational database (such as an emerging Bigtable data model), thereby realizing data services. Smooth migration.
  • the data migration in the embodiment of the present invention is automatic, not only ensures the reliability of migration, but also improves the efficiency of data migration, thereby realizing smooth switching of services.
  • FIG. 2 a flowchart of a data migration method according to Embodiment 2 of the present invention is provided.
  • the method includes:
  • Step 201 Add a SCHEMA definition to the non-relational database
  • the non-relational database takes the class Bigtable data as an example, but is not limited thereto. That is to say, the SCHEMA mapping engine first adds the SCHEMA definition to the non-relational database of the Bigtable data model, and the specific process is:
  • Step 201 Establish a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database, and record the mapping relationship defined by the SCHEMA in the mapping relationship table;
  • mapping relationship between the relational table and the non-relational table established by the SCHEMA mapping engine is as shown in Table 1 to Table 3:
  • Step 203 Obtain SCHEMA information from a relational database, and map the SCHEMA information into an XML model.
  • the SCHEMA mapping engine first requests the SCHEMA information from the relational database and maps it to the XML model.
  • mapping relational database tables to standard XML models there are more than 4 methods for mapping relational database tables to standard XML models.
  • .net, Delphi, etc. all provide automatic conversion methods, basically one-to-one mapping.
  • the method provided by the embodiment of the present invention is not limited.
  • the mapping conversion rule is:
  • Step 204 Mapping the XML model into a non-relational database XML model, and recording a corresponding mapping relationship in the mapping relationship table;
  • This embodiment provides a configuration interface, which can configure the default mapping mode of the SCHEMA mapping engine to be a normalized processing mode or a non-normalized processing mode; and can also implement a customizable mapping process and align the mapping results.
  • each table in the relational database is mapped to a column family of non-relational databases.
  • the SCHEMA mapping engine sequentially scans the XML files generated by direct mapping, and maps each database db node scanned to a non-relational database Keyspace node with default attribute definitions, and these attributes contain at least the number of copies. , copy distribution strategy; map each table node scanned in a db into a column family node, the column family column type should be determined according to the column SCHEMA definition in the table, if the column definition contains the union field (join primary key) ), the column type is "super", otherwise it is "standard”.
  • the necessary mode attributes must be added.
  • the above process can be implemented by the following procedures, but is not limited thereto.
  • Strategy_class org.apache.cassandra.locator.SimpleStrategy
  • ⁇ columnfamily name "ispace_msg_recvState”
  • the first segment xml program is a relation table xml file directly mapped into a relational database table
  • the second segment is a non-relational table xml file to which a default attribute value is mapped; mapping into a non-relational table xml file Then you can directly map to non-relational Keyspace and columnfamily, and create a non-relational table accordingly.
  • mapping relationship table is created before starting the mapping process for each database, and the mapping relationship between the relational table and the non-relational table in each db is recorded.
  • mapping relationship is recorded in the mapping table in the data migration device. As shown in Table 5, Table 5
  • the relational data model design should follow the principle of minimizing data redundancy, that is, normalization.
  • the public data is all extracted and stored in a separate table, and other tables are associated with it.
  • Non-relational databases use distributed servers to form distributed clusters, have flexible scalability, allow data redundancy, and in order to improve search efficiency, data that can be stored in a table is generally stored in a table, so the relationship is In the process of mapping database tables to non-relational database tables, - -
  • mapping relationship table is created before starting the mapping process for each database, and the mapping relationship between the relational table and the non-relational table in each db is recorded. Each time the SCHEMA mapping engine completes a table mapping, the mapping relationship is recorded in the established mapping table.
  • the SCHEMA mapping engine sequentially scans the XML files generated by direct mapping, and maps each database db node scanned to a non-relational database Keyspace node with default attribute definitions, and these attributes contain at least the number of copies.
  • the column type of the column family needs to be determined according to the column SCHEMA definition in the table. If the column definition includes a union field (join primary key), the column type is "super", otherwise it is "standard”.
  • the necessary mode attributes which include at least the column sort type and the cache configuration; - -
  • Strategy_class org.apache.cassandra.locator.SimpleStrategy
  • ⁇ columnfamily name "ispace_msg_recvState”
  • Step 205 Generate a data definition command according to the non-relational database XML model.
  • Step 206 Send the data definition command to a non-relational database, and create a non-relational database. - - SCHEMA;
  • Step 207 Obtain a data set from the relational database.
  • Step 208 Import the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.
  • mapping relationship in the record in the mapping relationship table is optimized:
  • mapping relationship table can also be modified through the configuration interface, so that the specified part is denormalized/normalized under the condition that most of the execution of the normalized/non-normalized mapping is guaranteed.
  • the default mode of the configuration data migration device is SCHEMA mapping mode, the SCHEMA mapping engine completes the automatic SCHEMA mapping, generates the non-relational database XML file, and fills in the mapping relationship table;
  • mapping relationship table specify the partial table to perform the denormalization (normalized processing mode) / normalization (in the non-normalized processing mode) processing mode, and then configure the data migration device to map and migrate the mode, the SCHEMA mapping engine starts the automatic mapping, Generating a non-relational database XML file according to the already filled mapping table;
  • the trigger executor performs data migration according to the mapping relationship.
  • the configuration interface provided by the data migration device can also modify the default attribute definitions of Keyspace and column family to be tuned according to the needs of the service.
  • the specific execution process is as follows:
  • the configuration data migration device is in the SCHEMA mapping mode, the SCHEMA mapping engine automatically completes the data model mapping, generates the non-relational database XML file, and fills in the mapping relationship table;
  • the trigger executor migrates data according to the mapping relationship.
  • SCHEMA is mapped to various modes.
  • - The process of shooting is described in detail, as shown in the figure, including:
  • Step 301 Determine whether it is a normalization mode; if yes, go to step 302; otherwise, go to step 303;
  • Step 302 Set the SCHEMA mapping engine to the normalization mode
  • Step 303 Set the SCHEMA mapping engine to a denormalized mode
  • Step 304 The SCHEMA mapping engine determines the mapping mode. If it is the SCHEMA mapping mode, step 305 is performed: if it is the mapping and data migration mode, step 310 is performed;
  • Step 305 Obtain SCHEMA information from a relational database, and directly map the SCHEMA information into a relational data XML model;
  • Step 306 Map the relational data XML model to a non-relational data model XML according to a specification mode and a mapping relationship table configuration;
  • Step 307 Determine whether there is a corresponding item in the mapping relationship table, and if yes, go to step 308; otherwise, go to step 309;
  • Step 308 Perform mapping according to the configuration in the correspondence table, and end;
  • Step 309 Perform mapping according to the configuration in the specification mode, and fill in the mapping relationship correspondence table, and end;
  • Step 310 Obtain SCHEMA information from the relational database, and directly map the SCHEMA information into a relational data XML model;
  • Step 311 Mapping the relational data XML model into a non-relational data model XML according to a specification mode and a mapping relationship table configuration;
  • Step 312 Determine whether there is a corresponding item in the mapping relationship table, and if yes, go to step 313; otherwise, go to step 314;
  • Step 313 Perform mapping according to the configuration in the correspondence table, and then perform step 315;
  • Step 314 Perform mapping according to the configuration in the specification mode, and fill in the mapping relationship correspondence table, and then perform step 315;
  • Step 315 Trigger the executor to perform the data migration process.
  • Figure 3 depicts the SCHEMA mapping process from a relational database to a non-relational database.
  • the data import process is performed by an executor in the data migration device, including:
  • the executor issues a data definition command to the non-relational database, and creates a SCHEMA information in the non-relational database according to the non-relational database XML model generated by the SCHEMA mapping process described above;
  • the actuator sequential scan directly maps each table of the XML file, and looks at the mapping relationship table to determine whether the table does not need to be mapped. If not, ignore the table and continue scanning the next table; if this table is the last table , then end; if mapping is required, perform step 3);
  • FIG. 4 is a flowchart of data import according to an embodiment of the present invention, which specifically includes:
  • Step 401 Trigger an executor to perform data migration
  • Step 402 Create a non-relational data table by calling a non-relational database interface according to the non-relational data model XML.
  • Step 403 sequentially scan a table table directly mapping XML, and view the mapping relationship table, determine whether the table Table does not need to be mapped; if yes, execute step 410; otherwise, perform step 404; Step 404: determine whether the table Table is associated Table, if yes, go to step 405; otherwise, go to step 406;
  • Step 405 Generate SQL that extracts data from the source table and its associated table, and extracts the data set from the relational database;
  • Step 406 Generate a source table, extract data SQL, and extract a data set from the relational database.
  • Step 407 Generate a data import function to the non-relational database according to the corresponding non-relational data model XML;
  • Step 408 Loop out the data strip from the data set, and use the import function to import the data strip;
  • Step 409 Determine whether there is still a table needs to be scanned, and if so, return to step 403; No, go to 'J, go to step 411;
  • Step 410 Determine whether there is still a table to scan, and if yes, return to step 403; Step 411;
  • a data migration device is also provided in the embodiment of the present invention.
  • the structure of the data migration device is as shown in FIG. 5.
  • the device includes: a first acquiring unit 51, a mapping unit 52, and a data migration unit 53,
  • the first obtaining unit 51 is configured to obtain the relational database table SCHEMA information from the relational database.
  • the mapping unit 52 is configured to map the SCHEMA information into a non-relational database XML model, and in the mapping relationship table. Recording the corresponding mapping relationship;
  • the data migration unit 53 is configured to import the data of the relational database into the non-relational database according to the mapping relationship recorded in the mapping relationship table.
  • the unit of the above device may be implemented by a processor, a memory or a special purpose circuit or the like.
  • the device may further include: a defining unit, connected to the first obtaining unit, configured to add a definition of the non-relational data object table SCHEMA in the non-relational database, specifically: adding a primary key to the column family SCHEMA constraint definition; adds SCHEMA constraint definitions for specific columns of the column family; maps one or more union primary key's relational database tables to super column families in non-relational databases.
  • a defining unit connected to the first obtaining unit, configured to add a definition of the non-relational data object table SCHEMA in the non-relational database, specifically: adding a primary key to the column family SCHEMA constraint definition; adds SCHEMA constraint definitions for specific columns of the column family; maps one or more union primary key's relational database tables to super column families in non-relational databases.
  • the device may further include: an establishing unit, connected to the mapping unit, configured to establish a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database.
  • an establishing unit connected to the mapping unit, configured to establish a mapping relationship table between the relational table in the relational database and the non-relational relational database in the non-relational database.
  • the first obtaining unit includes: a first sending unit and a first receiving unit, where the first sending unit is configured to send a request for acquiring SCHEMA information to the relational database; the first receiving unit And the SCHEMA information for receiving the feedback of the relational database.
  • the mapping unit includes: a first mapping unit and a second mapping unit, where the first mapping unit is configured to map the SCHEMA information into a relational database XML model; - a second mapping unit, configured to map the relational database XML model to a non-relational database XML model according to the definition of SCHEMA, and record the corresponding mapping relationship in the mapping relationship table.
  • the second mapping unit further includes: a normalization processing mode mapping unit and/or a non-normalization processing mode mapping unit, where the normalization processing mode mapping unit is configured to configure the relational database XML model into a normalized processing mode, and Mapping each table in the relational database to a column family in the non-relational database, and recording a corresponding mapping relationship in the mapping relationship table; the denormalized processing mode mapping unit is configured to configure the relational database XML model For the non-normalized processing mode, the tables in multiple relational databases are mapped to a column family in the non-relational database, and the corresponding mapping relationship is recorded in the mapping relation table.
  • the device may further include: a configuration optimization unit, configured to be connected to the second mapping unit, configured to be in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode The mapping relationship in the record is optimized.
  • a configuration optimization unit configured to be connected to the second mapping unit, configured to be in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode The mapping relationship in the record is optimized.
  • the first acquiring unit includes: a scanning unit, a generating unit, and a data acquiring unit, wherein the scanning unit is configured to scan a non-relational database XML model and a mapping relationship table to obtain a scan result; a generating unit, configured to generate an SQL query statement according to the scan result, where the data acquiring unit is configured to obtain a data set from the relational database by using the SQL query statement.
  • the data migration unit includes: a generating unit, a creating unit, a second obtaining unit, and a data importing unit, wherein the generating unit is configured to generate a data definition command according to the non-relational database XML model;
  • the creating unit is configured to send the data definition command to a non-relational database to create a SCHEMA in a non-relational database;
  • the second obtaining unit is configured to obtain a data set from the relational database;
  • the data And an importing unit configured to import the data set into the SCHEMA of the non-relational database according to the corresponding mapping relationship recorded in the mapping relationship table.
  • an embodiment of the present invention further provides a data migration system, and a schematic structural diagram thereof is shown in FIG. 6 .
  • the system comprises: at least one relational database (in the figure, the relational databases 61 and 62 are exemplified, but not limited thereto), and at least one non-relational database (in the figure, the non-relational databases 631 and 632 are taken as an example, but It is not limited to this and the data migration device 63.
  • the data migration device 63 is configured to obtain the relational database table SCHEMA information (such as the relational database 61) from the relational database; respectively map the SCHEMA information into a non-relational database XML model, and record correspondingly in the mapping relationship table.
  • SCHEMA information such as the relational database 61
  • mapping relationship data of a relational database (such as relational database 61) is imported into a non-relational database (such as non-relational database 631).
  • a relational database such as relational database 61
  • a non-relational database such as non-relational database 631.
  • the data of the non-relational database may be migrated to a plurality of non-relational databases, or the data in the plurality of relational databases may be migrated to a non-relational database, and the implementation process is similar. limit.
  • the data migration device in the system is the same as the data migration device in the foregoing embodiment, and is not described here.
  • the data migration apparatus in the embodiment of the present invention can implement the data migration requirement from the relational database to the non-relational database according to the modeling difference between the relational database and the non-relational database, and implements the multi-table from the relational database to the non-relational database.
  • Table mapping mechanism The traditional relational data model is designed to minimize data redundancy, that is, the principle of normalization. Public data is extracted and stored in a single table, and other tables are associated with it.
  • Non-relational databases use inexpensive servers to form distributed clusters, allowing data redundancy, and in order to improve search efficiency, data that can be integrated into a table is generally not stored in a table, so the mapping process from relational database tables to non-relational database tables In order to improve the operational efficiency of the data after migration, it is necessary to map multiple relational database tables into a non-relational database column family, that is, the principle of denormalization.
  • the apparatus of the embodiment of the present invention can automatically store the common table to the table where the foreign key is located. The operational efficiency of the data after migration and the convenient access mode are fully guaranteed.
  • FIG. 7 is a schematic structural diagram of a data migration system according to an embodiment of the present invention.
  • the system includes: at least one data relation library (in this embodiment, a data relation library is taken as an example) 71.
  • At least one non-data The relational library (this embodiment takes a non-data relational library as an example) 72, and the data migration means 73 respectively connected to the data relational library and the non-data relational library, but is not limited thereto.
  • the above means may be implemented by a processor, a memory or a special purpose circuit or the like.
  • the relational database 71 may be MySql, DB2, Oracle, etc., using a two-dimensional table to represent entities and their associations: using a table to represent a set of entities, a column to represent attributes, a table structure to represent a solid type ; Use specific redundancy information between tables to represent the connections between entities (primary keys, foreign keys).
  • the advantages are: integrity is easy to express; single, the concept of the table is intuitive, single, user-friendly; solid theoretical basis (relational algebra); non-procedural data request, request data can not specify the path; high data independence , users only need to ask "what to do", no need to explain "how to do”; support collection operations.
  • the non-relational database 72 may be a non-relational database, a cloud database, or a no sql database whose data model is a class Bigtable.
  • the non-relational database of the Bigtable data model is a cluster composed of inexpensive servers, and the cluster can self-organize and self-manage, and provide services as a whole.
  • Such databases are characterized by low cost, flexible scalability, mass storage, high concurrency, autonomous feature customization, and weak affinity.
  • the data migration device 73 may be a hardware device or a software device, and is used to implement a data mapping and migration process between the relational database and the non-relational database of the Bigtable data model. The specific process is to obtain the SCHEMA information from the relational database, map it to the non-relational database SCHEMA, and extract the data from the relational database according to the mapping relationship to import into the non-relational database.
  • the extracting the data from the relational database according to the mapping relationship into the non-relational database includes: generating a data definition command according to the non-relational database XML model; sending the data definition command to the non-relational database, creating SCHEMA in a non-relational database; obtaining a data set from the relational database; importing the data set into the SCHEMA of the non-relational database according to a corresponding mapping relationship recorded in the mapping relationship table.
  • the data migration device 73 includes: a first obtaining unit 731, configured to obtain a relational database table SCHEMA information from a relational database; the specific acquisition process is as described above, and details are not described herein; the SCHEMA mapping engine (ie, mapping) a unit 732, configured to map the SCHEMA information into a non-relational database XML model; an executor (ie, a data migration unit) 733, configured to import the data of the relational database into the non-relational database XML model according to the mapping relationship table in.
  • a first obtaining unit 731 configured to obtain a relational database table SCHEMA information from a relational database
  • the specific acquisition process is as described above, and details are not described herein
  • the SCHEMA mapping engine ie, mapping
  • a unit 732 configured to map the SCHEMA information into a non-relational database XML model
  • an executor ie, a data migration unit
  • the data migration device 73 may further include a mapping relationship table 730, which is established by the establishing unit; the mapping relationship table records mapping the SCHEMA information into a non-relational database XML - - The mapping relationship of the model.
  • the data migration device 73 may further include a definition unit for adding a non-relational data object table, a definition of SCHEMA, and the like in the non-relational database.
  • the device may further include: a configuration optimization unit, configured to optimize the mapping relationship in the record in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode, where the configuration may be specifically configured Interface 734.
  • a configuration optimization unit configured to optimize the mapping relationship in the record in the mapping relationship table when the relational database XML model is configured to be a normalized processing mode or a non-normalized processing mode, where the configuration may be specifically configured Interface 734.
  • the embodiment of the invention provides a method, a device and a system for data migration between a relational database and a non-relational database, which can be used for data migration between a traditional relational database and a non-relational database of a Bigtable data model.
  • the service can perform data migration in a reliable, fast, and convenient manner, smoothly transition to a non-relational database, and utilize a non-relational database for low cost, flexible scalability, and high High efficiency and other advantages enhance the competitiveness of the business.
  • the business data in the database can be switched to the non-relational database of the Bigtable data model at any time, and the entire data migration process is automatic, which also saves cost and reduces error rate, and realizes Smooth migration of business data.
  • the data migration device further provides a configuration interface, which can customize the data migration mode, and optimize the data table according to the service requirement, and also modify the mapping relationship table through the configuration interface, thereby adding more Attributes that implement more custom mappings.
  • This configurable data migration process enables the business to migrate data from a relational database to a non-relational database, and then migrate data according to the modeling characteristics of the non-relational database, giving full play to the advantages of the non-relational database and improving the migrated data. Access efficiency.
  • the present invention can be implemented by hardware, or can be implemented by means of software plus necessary general hardware platform.
  • the technical solution of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (for example, may be a read only memory (ROM), a USB flash drive, a mobile hard disk, a random access memory (RAM), a disk or an optical disk, etc., which can store program code, etc., including a plurality of instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to execute the present
  • ROM read only memory
  • USB flash drive a mobile hard disk
  • RAM random access memory
  • disk or an optical disk etc.
  • the disclosed systems, devices, and methods may be implemented in other ways without departing from the spirit and scope of the application.
  • the device embodiments described above are merely illustrative.
  • the division of the modules is only a logical function division.
  • there may be another division manner for example, multiple modules or components may be combined or Can be integrated into another system, or some features can be ignored, or not executed.
  • the modules described as separate components may or may not be physically separated.
  • the components displayed as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. . Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

Abstract

L'invention concerne un procédé, un dispositif et un système de migration de données. Le dispositif selon l'invention consiste : à obtenir d'une base de données relationnelle (101) des informations SCHEMA de table de base de données relationnelle ; à mapper les information SCHEMA sur un modèle XML de base de données non relationnelle et à enregistrer la relation de mappage correspondante dans une table de relation de mappage (102) ; à charger les données de la base de données relationnelle dans le modèle XML de base de données non relationnelle en fonction de la table de relation de mappage. La présente invention permet de régler le problème technique lié à une migration fluide des données de la base de données relationnelle dans la base de données non relationnelle et d'obtenir une conversion fluide de données de service.
PCT/CN2011/077102 2011-07-13 2011-07-13 Procédé, dispositif et système de migration de données WO2012083679A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN2011800012395A CN102308297B (zh) 2011-07-13 2011-07-13 一种数据迁移方法、数据迁移装置及数据迁移系统
PCT/CN2011/077102 WO2012083679A1 (fr) 2011-07-13 2011-07-13 Procédé, dispositif et système de migration de données

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2011/077102 WO2012083679A1 (fr) 2011-07-13 2011-07-13 Procédé, dispositif et système de migration de données

Publications (1)

Publication Number Publication Date
WO2012083679A1 true WO2012083679A1 (fr) 2012-06-28

Family

ID=45381255

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2011/077102 WO2012083679A1 (fr) 2011-07-13 2011-07-13 Procédé, dispositif et système de migration de données

Country Status (2)

Country Link
CN (1) CN102308297B (fr)
WO (1) WO2012083679A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10002142B2 (en) 2012-09-29 2018-06-19 International Business Machines Corporation Method and apparatus for generating schema of non-relational database
CN112783856A (zh) * 2019-11-07 2021-05-11 北京沃东天骏信息技术有限公司 数据迁移方法和装置
TWI740899B (zh) * 2016-03-22 2021-10-01 香港商阿里巴巴集團服務有限公司 資料遷移的最佳化方法、評估方法及處理方法及裝置

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103514201B (zh) * 2012-06-27 2017-05-03 阿里巴巴集团控股有限公司 一种非关系型数据库的数据查询方法和装置
CN103455589B (zh) * 2013-08-29 2016-09-21 中国农业银行股份有限公司 产品工厂模式下的产品数据迁移方法、装置及系统
CN103514274A (zh) * 2013-09-17 2014-01-15 宁波东冠科技有限公司 非关系型数据库HBase的数据迁移方法
CN104657396B (zh) 2013-11-25 2020-04-24 腾讯科技（深圳）有限公司 数据迁移方法及装置
CN103810275B (zh) * 2014-02-13 2018-02-09 清华大学 用于非关系与关系型数据库间数据交互的方法和装置
CN105095237B (zh) * 2014-04-30 2018-07-17 国际商业机器公司 用于生成非关系数据库的模式的方法和设备
CN104123392A (zh) * 2014-08-11 2014-10-29 吉林禹硕动漫游戏科技股份有限公司 一种关系型数据库迁移到HBase数据库的工具和方法
KR101674924B1 (ko) * 2014-09-26 2016-11-22 삼성에스디에스 주식회사 데이터베이스 마이그레이션 방법 및 그 장치
CN105574042A (zh) * 2014-10-17 2016-05-11 中国移动通信集团公司 一种电子健康档案数据的处理方法及装置
US10409835B2 (en) * 2014-11-28 2019-09-10 Microsoft Technology Licensing, Llc Efficient data manipulation support
CN104504008B (zh) * 2014-12-10 2018-10-02 华南师范大学 一种基于嵌套的SQL到HBase的数据迁移算法
US20160188710A1 (en) * 2014-12-29 2016-06-30 Wipro Limited METHOD AND SYSTEM FOR MIGRATING DATA TO NOT ONLY STRUCTURED QUERY LANGUAGE (NoSOL) DATABASE
CN105808539A (zh) * 2014-12-29 2016-07-27 北京航天测控技术有限公司 一种数据导出导入方法及装置
CN105373599B (zh) * 2015-10-28 2017-05-31 北京汇商融通信息技术有限公司 基于多种数据存储平台的数据迁移系统
CN105550263A (zh) * 2015-12-05 2016-05-04 百势软件(北京)有限公司 一种对象关系映射中的赋值方法及装置
CN106933859B (zh) * 2015-12-30 2020-10-20 中国移动通信集团公司 一种医疗数据的迁移方法和装置
CN105938479B (zh) * 2016-03-31 2019-10-22 华南师范大学 一种关系表与非关系表的结构转换方法
CN106503040B (zh) * 2016-09-20 2019-08-02 福建天晴数码有限公司 适用sql查询方法的kv数据库及其创建方法
TW201820175A (zh) * 2016-11-22 2018-06-01 財團法人資訊工業策進會 資料庫轉換伺服器及其資料庫轉換方法
CN108228645A (zh) * 2016-12-21 2018-06-29 凌群电脑股份有限公司 大数据资料库系统
CN107689949B (zh) 2017-03-31 2020-03-17 平安科技（深圳）有限公司 数据库权限管理方法和系统
CN109660576B (zh) * 2017-10-10 2021-09-07 武汉斗鱼网络科技有限公司 用户数据实时迁移方法、存储介质、电子设备及系统
CN107807977B (zh) * 2017-10-26 2020-04-03 河海大学 一种基于配置的对象属性元数据抽取系统
CN110555012B (zh) * 2018-05-14 2022-03-25 杭州海康威视数字技术股份有限公司 数据迁移方法及装置
CN109388670A (zh) * 2018-09-18 2019-02-26 重庆航天职业技术学院 一种xml与关系数据库的转换方法
CN110580246B (zh) * 2019-07-30 2023-10-20 平安科技（深圳）有限公司 迁徙数据的方法、装置、计算机设备及存储介质
CN110716913B (zh) * 2019-07-31 2022-12-20 安徽四创电子股份有限公司 一种Kafka与Elasticsearch数据库数据的互相迁移方法
CN110674113A (zh) * 2019-09-24 2020-01-10 咪咕音乐有限公司 数据的一键迁移方法、装置、电子设备及存储介质
CN113901031A (zh) * 2021-10-21 2022-01-07 掌阅科技股份有限公司 数据库迁移方法、电子设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1687927A (zh) * 2005-05-08 2005-10-26 浙江大学 一种实现信息系统数据自动交换的方法
CN1815468A (zh) * 2005-01-31 2006-08-09 微软公司 非关系型查询语言与关系型数据存储的集成
CN101697166A (zh) * 2009-10-28 2010-04-21 浪潮电子信息产业股份有限公司 一种加速异构平台数据整合的方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7103611B2 (en) * 2003-05-01 2006-09-05 Oracle International Corporation Techniques for retaining hierarchical information in mapping between XML documents and relational data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1815468A (zh) * 2005-01-31 2006-08-09 微软公司 非关系型查询语言与关系型数据存储的集成
CN1687927A (zh) * 2005-05-08 2005-10-26 浙江大学 一种实现信息系统数据自动交换的方法
CN101697166A (zh) * 2009-10-28 2010-04-21 浪潮电子信息产业股份有限公司 一种加速异构平台数据整合的方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10002142B2 (en) 2012-09-29 2018-06-19 International Business Machines Corporation Method and apparatus for generating schema of non-relational database
TWI740899B (zh) * 2016-03-22 2021-10-01 香港商阿里巴巴集團服務有限公司 資料遷移的最佳化方法、評估方法及處理方法及裝置
CN112783856A (zh) * 2019-11-07 2021-05-11 北京沃东天骏信息技术有限公司 数据迁移方法和装置

Also Published As

Publication number Publication date
CN102308297A (zh) 2012-01-04
CN102308297B (zh) 2013-06-05

Similar Documents

Publication Publication Date Title
WO2012083679A1 (fr) Procédé, dispositif et système de migration de données
US11520780B2 (en) Distributed database systems and structures
US11281531B2 (en) Serial storage node processing of data functions
TWI710919B (zh) 資料儲存裝置、轉譯裝置及資料庫存取方法
US10769148B1 (en) Relocating data sharing operations for query processing
JP6602355B2 (ja) クラウドベースの分散永続性及びキャッシュデータモデル
CN111259006B (zh) 一种通用的分布式异构数据一体化物理汇聚、组织、发布与服务方法及系统
CN103312791B (zh) 物联网异构数据存储方法及系统
US10642837B2 (en) Relocating derived cache during data rebalance to maintain application performance
WO2016180055A1 (fr) Procédé, dispositif et système de stockage et de lecture de données
CN104462185A (zh) 一种基于混合结构的数字图书馆云存储系统
CN103078898B (zh) 文件系统、接口服务装置和数据存储服务提供方法
CN104050276A (zh) 一种分布式数据库的缓存处理方法及系统
US20210271658A1 (en) Data edge platform for improved storage and analytics
CN105069151A (zh) HBase二级索引构建装置和方法
CN108427728A (zh) 元数据的管理方法、设备及计算机可读介质
US20230376475A1 (en) Metadata management method, apparatus, and storage medium
JP2020525938A (ja) データベース内でテナントを作成及び削除するシステム及び方法
US10990324B2 (en) Storage node processing of predefined data functions
CN105022779A (zh) 一种利用Filesystem API实现HDFS文件存取方法
CN109413130A (zh) 一种云存储系统
WO2014180395A1 (fr) Procédé et système de stockage de fusion de données massives
US9292523B1 (en) Managing data storage
WO2024011855A1 (fr) Procédé et appareil permettant de former un modèle de classification de données, dispositif, et support de stockage
CN116627448A (zh) 一种创建微服务的方法及相关设备

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201180001239.5

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11852082

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11852082

Country of ref document: EP

Kind code of ref document: A1