CN115617773A - Data migration method, device and system - Google Patents

Data migration method, device and system Download PDF

Info

Publication number
CN115617773A
CN115617773A CN202110806666.6A CN202110806666A CN115617773A CN 115617773 A CN115617773 A CN 115617773A CN 202110806666 A CN202110806666 A CN 202110806666A CN 115617773 A CN115617773 A CN 115617773A
Authority
CN
China
Prior art keywords
target
source
database
field
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110806666.6A
Other languages
Chinese (zh)
Inventor
谭晓敏
徐雄
张玉忠
杨豪杰
方艾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Telecom Corp Ltd
Original Assignee
China Telecom Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Telecom Corp Ltd filed Critical China Telecom Corp Ltd
Priority to CN202110806666.6A priority Critical patent/CN115617773A/en
Publication of CN115617773A publication Critical patent/CN115617773A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/211—Schema design and management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/445—Program loading or initiating
    • G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files

Abstract

The disclosure provides a method, a device and a system for data migration, and relates to the technical field of computers. In the method, a first mapping rule for data migration from a source data type of a source database to a target data type of a target database is obtained; converting the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule; determining target configuration information of corresponding fields in a target database according to the source configuration information of each field in each data table of the source database and the configuration capacity of the target data type of the target database; and generating a target statement for data migration according to the determined target data type and the target configuration information, and executing the target statement to complete the data migration from the source database to the target database. The method and the system can realize automatic data migration, ensure the consistency of the data before and after migration, save labor cost and improve the efficiency and accuracy of data migration.

Description

Data migration method, device and system
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, and a system for data migration.
Background
Various types of databases or search engines for storing data, such as MYSQL database, oracle database, SQL Server database, SQLite database, noSQL database, newSQL database, search engines, etc., have emerged with the development of big data technologies, making data migration technology between various databases, especially between legacy databases and new databases, an increasingly common need. Data migration refers to migrating data in a source database to a target database, and is data synchronization between databases or between various heterogeneous databases. Data migration needs to ensure data synchronization between multiple data platforms or databases, that is, to ensure consistency before and after data migration. Data migration also gradually has wider and wider application scenarios, such as storage and analysis of business data, upgrading of new systems, and the like.
In some related technologies, a Database schema Definition Language (DDL) statement of a target Database is manually written by a Database Administrator (DBA) to implement Data migration.
Disclosure of Invention
In the related art, a database administrator needs to spend a lot of time to modify the repetitive work of the DDL statements such as table building, which is inefficient.
To this end, the present disclosure provides a method of data migration that can improve efficiency.
In the embodiment of the disclosure, a first mapping rule for data migration from a source data type of a source database to a target data type of a target database is obtained; converting the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule; determining target configuration information of corresponding fields in a target database according to the source configuration information of each field in each data table of the source database and the configuration capacity of the target data type of the target database; and generating a target statement for data migration according to the determined target data type and the target configuration information, and executing the target statement to complete the data migration from the source database to the target database. The method and the device can realize automatic data migration, ensure the consistency of data before and after migration, save labor cost and improve the efficiency and accuracy of data migration.
According to some embodiments of the present disclosure, there is provided a method of data migration, comprising:
acquiring a first mapping rule for data migration from a source data type of a source database to a target data type of a target database;
converting the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule;
determining target configuration information of corresponding fields in a target database according to the source configuration information of each field in each data table of the source database and the configuration capacity of the target data type of the target database;
and generating a target statement for data migration according to the determined target data type and the target configuration information, and executing the target statement to complete the data migration from the source database to the target database.
In some embodiments, the determining the target configuration information of the corresponding field in the target database comprises:
judging whether the source configuration information of each field conforms to the configuration capability of the target data type of the target database;
under the condition that the source configuration information of the source data type of the selected field conforms to the configuration capability of the target data type, taking the source configuration information of the source data type of the selected field as the target configuration information of the target data type;
in case the source configuration information of the source data type of the selected field does not comply with the configuration capability of the target data type, the following steps are performed: acquiring a second mapping rule for carrying out data migration from the source configuration information of the source data type of the selected field to the target configuration information of the target data type; and converting the source configuration information of the source data type of the selected field into target configuration information supported by the target data type of the target database according to the second mapping rule.
In some embodiments, further comprising:
for each field in the data table of the source database, performing the following steps:
judging whether the field is a designated field;
under the condition that the field is the designated field, carrying out data migration on the designated field according to a third mapping rule of the designated field;
and selecting the target data type and the target configuration information which are matched with the specified field in the target database for the specified field by the third mapping rule.
In some embodiments, further comprising:
generating a first mapping rule, and adding the first mapping rule to a mapping rule base;
the generating a first mapping rule comprises:
acquiring source data types of all fields of a source database and candidate data types supported by a target database;
for each field in the data table of the source database, performing the following steps: selecting a target data type which is most matched with the source data type of the field from candidate data types supported by a target database; establishing a mapping relation from a source data type of the source database to a target data type of the target database;
and taking the mapping relations corresponding to the source data types of all the fields in the data table of the source database as a first mapping rule for carrying out data migration from the source data type of the source database to the target data type of the target database.
In some embodiments, further comprising:
generating a second mapping rule, and adding the second mapping rule to a mapping rule base;
the generating a second mapping rule comprises:
acquiring source configuration information of each field of a source database and configuration capacity of a target data type of a target database;
for each field in the data table of the source database, performing the following steps: selecting target configuration information of the target data type which is most matched with the source configuration information of the field according to the configuration capability of the target data type of the target database; establishing a mapping relation from the source configuration information of the field of the source database to the target configuration information of the target data type of the target database;
and taking the mapping relations corresponding to the source configuration information of all fields in the data table of the source database as a second mapping rule for carrying out data migration from the source configuration information of the source database to the target configuration information of the target database.
In some embodiments, further comprising:
generating a third mapping rule, and adding the third mapping rule to a mapping rule base;
the generating the third mapping rule comprises:
selecting a target data type and target configuration information which are most matched with the specified fields from a target database;
establishing a mapping relation from the source data type and the source configuration information of the designated field of the source database to the target data type and the target configuration information of the target database;
and taking the mapping relation corresponding to the specified field as a third mapping rule for data migration of the specified field from the source database to the target database.
In some embodiments, the determining whether the source configuration information of each field satisfies the configuration capability of the target data type of the target database includes:
judging whether one or more items of data length, data precision or constraint of each field meet the configuration capability of the target data type of the target database;
wherein the constraints comprise one or more of a primary key constraint with a field as a primary key, a constraint with a field as a unique value, a check constraint with a field as a specified value, a default constraint with a field as a default value, and a foreign key constraint with a field as a foreign key.
According to further embodiments of the present disclosure, there is provided an apparatus for data migration, including:
the acquisition module is configured to acquire a first mapping rule for data migration from a source data type of a source database to a target data type of a target database;
the conversion module is configured to convert the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule;
the determining module is configured to determine target configuration information of corresponding fields in the target database according to the source configuration information of each field in each data table of the source database and the configuration capability of the target data type of the target database;
and the migration module is configured to generate a target statement for data migration according to the determined target data type and the target configuration information and execute the target statement so as to complete data migration from the source database to the target database.
According to still further embodiments of the present disclosure, there is provided an apparatus for data migration, including: a memory; and a processor coupled to the memory, the processor configured to perform the method of data migration of any embodiment based on instructions stored in the memory.
According to still further embodiments of the present disclosure, there is provided a system for data migration, including: the source database is used for providing source data types and source configuration information of the fields; the target database is used for providing candidate data types supported by the target database and corresponding configuration capacity; and, the apparatus for data migration of any embodiment.
According to still further embodiments of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of data migration of any of the embodiments.
Drawings
The drawings that will be used in the description of the embodiments or the related art will be briefly described below. The present disclosure can be understood more clearly from the following detailed description, which proceeds with reference to the accompanying drawings.
It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without undue inventive faculty.
FIG. 1 illustrates a flow diagram of a method of data migration in accordance with some embodiments of the present disclosure.
FIG. 2 illustrates a schematic diagram of an apparatus for data migration, in accordance with some embodiments of the present disclosure.
FIG. 3 illustrates a schematic diagram of an apparatus for data migration in accordance with further embodiments of the present disclosure.
FIG. 4 illustrates a schematic diagram of a system for data migration, according to some embodiments of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure.
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The description of the exemplary embodiments is merely illustrative and is in no way intended to limit the disclosure, its application, or uses. The present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. It should be noted that: the relative arrangement of parts and steps, the composition of materials and values set forth in these embodiments are to be construed as illustrative only and not as limiting unless otherwise specifically stated.
The use of the word "comprising" or "comprises" and the like in this disclosure means that the elements listed before the word encompass the elements listed after the word and do not exclude the possibility that other elements may also be encompassed.
All terms (including technical or scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs unless specifically defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
The descriptions of "first", "second", "third", etc. in this disclosure are intended to refer to different objects unless otherwise specified, and are not intended to refer to the meaning of size or timing, etc.
FIG. 1 illustrates a flow diagram of a method of data migration in accordance with some embodiments of the present disclosure. The method may be performed, for example, by an apparatus for data migration.
As shown in FIG. 1, the method of this embodiment includes steps 110-140.
In step 110, a first mapping rule for data migration from a source data type of a source database to a target data type of a target database is obtained.
The source database or the target database may be, for example, a conventional database, including but not limited to MySQL, oracle, postgreSQL; it may also be a new database, including but not limited to, tedb, elasticsearch; but also enterprise self-research databases. The TiDB is a fusion database product positioned in online transaction/online analysis Processing (HTAP), and the elastic search is a search server based on Lucene and provides a full-text search engine with distributed multi-user capability.
Wherein the source data type or the target data type includes, but is not limited to, integer, floating point, character, text, binary string, date, timestamp, and the like.
In some embodiments, further comprising: and generating a first mapping rule and adding the first mapping rule to the mapping rule base. Generating the first mapping rule includes: acquiring source data types of all fields of a source database and candidate data types supported by a target database; for each field in the data table of the source database, performing the following steps: selecting a target data type which is most matched with the source data type of the field from candidate data types supported by the target database, and establishing a mapping relation from the source data type of the source database to the target data type of the target database; and taking the mapping relation corresponding to the source data types of all fields in the data table of the source database as a first mapping rule for carrying out data migration from the source data type of the source database to the target data type of the target database.
The first mapping rule is that a target data type which is closest to and most matched with a source data type is selected from a target database, and data migration is performed according to the target data type, so that the problem that the data types of heterogeneous databases are inconsistent is effectively solved, the consistency of the data before and after the data migration is ensured, and the success rate of the data migration can be improved.
In step 120, the source data type of each field in the data tables of the source database is converted into the target data type of the corresponding field in the target database according to the first mapping rule.
For example, the source database is a MySQL database, the target database is an Oracle database, and a data table in the MySQL database has three fields of DEVICE _ ID, DOMAIN _ NAME and IP. The corresponding data storage format of the DEVICE _ ID field in the MySQL database is decimal (24, 0), which means that the data type of the DEVICE _ ID field is decimal, the data length is 24, and the data precision is 0 bit after the decimal point. The corresponding data storage format of the DOMAIN _ NAME field in the MySQL database is varchar (64), the data type of the DOMAIN _ NAME field is varchar, and the data length is 64. The corresponding data storage format of the IP field in the MySQL database is varchar (255), which indicates that the data type of the IP field is varchar and the data length is 255.
According to the first mapping rule, the target data type that is most matched with the decimall data type in the Oracle database is the number data type, and the target data type that is most matched with the Varchar data type in the Oracle database is the Varchar2 data type.
Mapping each field to a field in the target database Oracle database respectively as shown in the following table:
name of field MySQL database Oracle database
DEVICE_ID decimal(24,0) number(24)
DOMAIN_NAME varchar(64) Varchar2(64)
IP varchar(255) Varchar2(255)
For another example, assume that the source database is a MySQL database and the target database is a target search engine ES. According to the first mapping rule, the target data type which is matched with the decimal data type in the ES database is the double data type, and the target data type which is matched with the varchar data type in the ES database is the text data type.
Mapping each field to a field in the target database ES respectively as shown in the following table:
name of field MySQL database ES database
DEVICE_ID decimal(24,0) double
DOMAIN_NAME varchar(64) text
IP varchar(255) text
The source data type of the source database is converted into the target data type which is closest to and most matched with the source data type in the target database through the first mapping rule, so that the success rate of data migration can be improved, and the data migration can be efficiently and automatically completed.
In step 130, the target configuration information of the corresponding field in the target database is determined according to the source configuration information of each field in each data table of the source database and the configuration capability of the target data type of the target database.
In some embodiments, determining the target configuration information for the corresponding field at the target database comprises:
firstly, judging whether the source configuration information of each field accords with the configuration capability of the target data type of the target database.
And under the condition that the source configuration information of the source data type of the selected field conforms to the configuration capability of the target data type, taking the source configuration information of the source data type of the selected field as the target configuration information of the target data type.
Suppose that the source configuration information of the source database MySQL database is mapped to the target database Oracle database, and the target configuration information of the corresponding field in the target database Oracle database is determined. For example, the DEVICE _ ID field, default (24, 0), data length 24 does not exceed the NUMBER maximum accuracy, satisfying the data accuracy requirement; the DOMAIN _ NAME field, varchar (64), has a data length of 64 characters at most, and does not exceed the configuration capability of Varchar2, i.e. the maximum number of characters that Varchar2 can support, and meets the data length requirement. The IP field, varchar (255), has a data length of up to 255 characters and does not exceed the configuration capability of Varchar2, i.e., the maximum number of characters that Varchar2 can support.
In case the source configuration information of the source data type of the selected field does not comply with the configuration capability of the target data type, the following steps are performed: acquiring a second mapping rule for data migration from the source configuration information of the source data type of the selected field to the target configuration information of the target data type; and converting the source configuration information of the source data type of the selected field into target configuration information supported by the target data type of the target database according to the second mapping rule.
For example, the source database is MySQL database and the target database is search engine ES database, DEVICE _ ID field, default (24, 0), and the data precision 24 exceeds the configuration capability of the DOUBLE data type in the ES database, i.e. exceeds the maximum value 15 of the data precision that can be supported by the DOUBLE data type, and cannot meet the requirement that the data precision required by the DEVICE _ ID field is 24. Thus, according to the second mapping rule, when decimal (M, D), where M >15, this field maps to a TEXT type. The DEVICE _ ID field is mapped to the ES database according to a second mapping rule, as shown in the following table:
name of field MySQL database ES database
DEVICE_ID decimal(24,0) text
And converting the source data type of the source database into the target data type which is closest to and most matched with the source data type in the target database through a second mapping rule, so that the success rate of data migration can be improved, and the data migration can be efficiently and automatically completed.
For example, determining whether the source configuration information of each field satisfies the configuration capability of the target data type of the target database may include determining whether one or more of a data length, a data precision, or a constraint of each field satisfies the configuration capability of the target data type of the target database. The constraint includes one or more of a primary Key constraint (primary Key constraint) whose field is a primary Key, a constraint (Unique constraint) whose field is a Unique value, a Check constraint (Check constraint) whose field is a specified value, a Default constraint (Default constraint) whose field is a Default value, and an external Key constraint (Foreign Key constraint) whose field is an external Key. Wherein, the field of the primary key constraint has uniqueness and non-null; the field of the unique constraint has uniqueness, can be empty, but only has one; the fields of the examination constraints have limitations on the range, format (e.g., age, gender, etc.) of the column data; the field of the default constraint has a default value that specifies data (e.g., the default constraint of the gender field is set to a specified value of "male"); the fields of the foreign key constraint need to establish a relationship between the two data tables and reference the specified fields of the primary data table.
In some embodiments, further comprising: and generating a second mapping rule and adding the second mapping rule to the mapping rule base.
Wherein generating the second mapping rule comprises: acquiring source configuration information of each field of a source database and configuration capacity of a target data type of a target database; for each field in the data table of the source database, performing the following steps: selecting target configuration information of the target data type which is most matched with the source configuration information of the field according to the configuration capability of the target data type of the target database, and establishing a mapping relation from the source configuration information of the field of the source database to the target configuration information of the target data type of the target database; and taking the mapping relations corresponding to the source configuration information of all fields in the data table of the source database as a second mapping rule for carrying out data migration from the source configuration information of the source database to the target configuration information of the target database.
The second mapping rule is that the target configuration information which is closest to and most matched with the source configuration information of the field is selected according to the target data type in the target database, and data migration is carried out according to the target configuration information, so that the success rate of data migration can be further improved.
In some embodiments, further comprising: for each field in the data table of the source database, performing the following steps: judging whether the field is a designated field; under the condition that the field is the designated field, carrying out data migration of the designated field according to a third mapping rule of the designated field; and selecting the target data type and the target configuration information which are matched with the specified field in the target database for the specified field by the third mapping rule.
For example, the source database is a MySQL database and the target database is a search engine ES database, and the target data type and the target configuration information that are the best match of the IP field in the ES database are the IP data type and the corresponding configuration capability thereof. Thus, according to the third mapping rule, the IP data type in the target database ES database is matched for the specified field IP field. Mapping the IP field to the ES database according to a third mapping rule, as shown in the following table:
name of field MySQL database ES database
IP varchar(255) IP
And selecting the target data type and the target configuration information which are most matched with the specified field in the target database for the specified field through the third mapping rule, so that the special service requirement of the specified field can be met, the accuracy of data migration is improved, and the data migration is efficiently and automatically completed.
In some embodiments, further comprising: and generating a third mapping rule and adding the third mapping rule to the mapping rule base. Generating the third mapping rule includes: selecting a target data type and target configuration information which are most matched with the specified field from a target database; establishing a mapping relation from a source data type and source configuration information of a designated field of a source database to a target data type and target configuration information of a target database; and taking the mapping relation corresponding to the designated field as a third mapping rule for performing data migration on the designated field from the source database to the target database.
The third mapping rule is that the target data type and the target configuration information which are closest to and most matched with the specified field are selected in the target database, and data migration is carried out according to the target data type and the target configuration information, so that the success rate of data migration can be further improved.
In step 140, according to the determined target data type and the target configuration information, a target statement for data migration is generated and executed to complete the data migration from the source database to the target database.
For example, according to the determined target data type and target configuration information, automatically generating a target DDL statement for data migration, such as creating a database, creating a data table, wherein the data type and configuration information of the fields in the data table are determined according to the target data type and target configuration information determined in the above embodiments.
In the embodiment, by determining the target data type and the target configuration information of each field, automatic data migration between different databases can be realized, the consistency of data of heterogeneous databases before and after migration is ensured, the labor cost is saved, the efficiency of data migration is improved, and the accuracy and the success rate of data migration can be improved.
FIG. 2 illustrates a schematic diagram of an apparatus for data migration, according to some embodiments of the present disclosure.
As shown in fig. 2, the apparatus of this embodiment includes an obtaining module 210, a converting module 220, a determining module 230, and a migrating module 240.
The obtaining module 210 is configured to obtain a first mapping rule for data migration from a source data type of a source database to a target data type of a target database.
In some embodiments, further comprising: and generating a first mapping rule and adding the first mapping rule to the mapping rule base. Generating the first mapping rule includes: acquiring source data types of all fields of a source database and candidate data types supported by a target database; for each field in the data table of the source database, performing the following steps: selecting a target data type which is most matched with the source data type of the field from the candidate data types supported by the target database, and establishing a mapping relation from the source data type of the source database to the target data type of the target database; and taking the mapping relation corresponding to the source data types of all fields in the data table of the source database as a first mapping rule for carrying out data migration from the source data type of the source database to the target data type of the target database.
The converting module 220 is configured to convert the source data type of each field in the data tables of the source database into the target data type of the corresponding field in the target database according to the first mapping rule.
The determining module 230 is configured to determine the target configuration information of the corresponding field in the target database according to the source configuration information of each field in the data tables of the source database and the configuration capability of the target data type of the target database.
In some embodiments, determining the target configuration information for the corresponding field in the target database comprises:
firstly, judging whether the source configuration information of each field accords with the configuration capability of the target data type of the target database. For example, one or more of the data length, data precision or constraints of each field is determined whether the configuration capability of the target data type of the target database is satisfied. Wherein the constraint comprises one or more of a primary key constraint with a field as a primary key, a constraint with a field as a unique value, a check constraint with a field as a specified value, a default constraint with a field as a default value, and a foreign key constraint with a field as a foreign key.
And under the condition that the source configuration information of the source data type of the selected field conforms to the configuration capability of the target data type, taking the source configuration information of the source data type of the selected field as the target configuration information of the target data type.
In case the source configuration information of the source data type of the selected field does not comply with the configuration capability of the target data type, the following steps are performed: acquiring a second mapping rule for data migration from the source configuration information of the source data type of the selected field to the target configuration information of the target data type; and converting the source configuration information of the source data type of the selected field into target configuration information supported by the target data type of the target database according to the second mapping rule.
In some embodiments, further comprising: and generating a second mapping rule and adding the second mapping rule to the mapping rule base. Generating the second mapping rule includes: acquiring source configuration information of each field of a source database and configuration capacity of a target data type of a target database; for each field in the data table of the source database, performing the following steps: selecting target configuration information of the target data type which is most matched with the source configuration information of the field according to the configuration capability of the target data type of the target database, and establishing a mapping relation from the source configuration information of the field of the source database to the target configuration information of the target data type of the target database; and taking the mapping relations corresponding to the source configuration information of all fields in the data table of the source database as a second mapping rule for carrying out data migration from the source configuration information of the source database to the target configuration information of the target database.
In some embodiments, further comprising: for each field in the data table of the source database, performing the following steps: judging whether the field is a designated field; under the condition that the field is the designated field, carrying out data migration of the designated field according to a third mapping rule of the designated field; and selecting the target data type and the target configuration information which are most matched with the specified field in the target database for the specified field by the third mapping rule.
In some embodiments, further comprising: and generating a third mapping rule and adding the third mapping rule to the mapping rule base. Generating the third mapping rule includes: selecting a target data type and target configuration information which are most matched with the specified field from a target database; establishing a mapping relation from a source data type and source configuration information of a designated field of a source database to a target data type and target configuration information of a target database; and taking the mapping relation corresponding to the specified field as a third mapping rule for performing data migration on the specified field from the source database to the target database.
And the migration module 240 is configured to generate a target statement for data migration according to the determined target data type and the target configuration information and execute the target statement to complete the data migration from the source database to the target database.
According to the data migration device in the embodiment, the target data type and the target configuration information of each field are determined, so that automatic data migration between different databases can be realized, the consistency of data of heterogeneous databases before and after migration is ensured, the labor cost is saved, the data migration efficiency is improved, and meanwhile, the accuracy and the success rate of data migration can be improved.
FIG. 3 shows a schematic diagram of an apparatus for data migration, according to further embodiments of the present disclosure.
As shown in fig. 3, the apparatus 300 for data migration of this embodiment includes: a memory 310 and a processor 320 coupled to the memory 310, the processor 320 configured to perform a method of data migration in any of the embodiments of the present disclosure based on instructions stored in the memory 310.
For example, a first mapping rule for performing data migration from a source data type of a source database to a target data type of a target database is obtained; converting the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule; determining target configuration information of corresponding fields in a target database according to the source configuration information of each field in each data table of the source database and the configuration capacity of the target data type of the target database; and generating a target statement for data migration according to the determined target data type and the target configuration information, and executing the target statement to complete the data migration from the source database to the target database.
Memory 310 may include, for example, system memory, fixed non-volatile storage media, and the like. The system memory stores, for example, an operating system, an application program, a Boot Loader (Boot Loader), and other programs.
The apparatus 300 for data migration may further include an input output interface 330, a network interface 340, a storage interface 350, and the like. These interfaces 330, 340, 350 and the memory 310 and the processor 320 may be connected, for example, by a bus 360. The input/output interface 330 provides a connection interface for input/output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 340 provides a connection interface for various networking devices. The storage interface 350 provides a connection interface for external storage devices such as an SD card and a usb disk.
FIG. 4 illustrates a schematic diagram of a system for data migration, according to some embodiments of the present disclosure.
As shown in fig. 4, the apparatus 400 for data migration of this embodiment includes: a source database 410, a data migration initiator 420, and a target database 430.
The source database 410 is used to provide source data type and source configuration information of the fields. And a target database 430 for providing candidate data types and corresponding configuration capabilities supported by the target database. The data migration apparatus 420, which may be, for example, the data migration apparatus described in any of the embodiments of fig. 2 or fig. 3, is configured to perform the method of data migration in any of the embodiments of the present disclosure.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-non-transitory readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is meant to be illustrative of the preferred embodiments of the present disclosure and not to be taken as limiting the disclosure, and any modifications, equivalents, improvements and the like that are within the spirit and scope of the present disclosure are intended to be included therein.

Claims (11)

1. A method of data migration, comprising:
acquiring a first mapping rule for data migration from a source data type of a source database to a target data type of a target database;
converting the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule;
determining target configuration information of corresponding fields in a target database according to the source configuration information of each field in each data table of the source database and the configuration capacity of the target data type of the target database;
and generating a target statement for data migration according to the determined target data type and the target configuration information, and executing the target statement to complete the data migration from the source database to the target database.
2. The method of data migration according to claim 1, wherein the determining the target configuration information of the corresponding field in the target database comprises:
judging whether the source configuration information of each field conforms to the configuration capability of the target data type of the target database;
under the condition that the source configuration information of the source data type of the selected field conforms to the configuration capability of the target data type, taking the source configuration information of the source data type of the selected field as the target configuration information of the target data type;
in case the source configuration information of the source data type of the selected field does not comply with the configuration capability of the target data type, the following steps are performed: acquiring a second mapping rule for carrying out data migration from the source configuration information of the source data type of the selected field to the target configuration information of the target data type; and converting the source configuration information of the source data type of the selected field into target configuration information supported by the target data type of the target database according to the second mapping rule.
3. The method of data migration of claim 1, further comprising:
for each field in the data table of the source database, performing the following steps:
judging whether the field is a designated field;
under the condition that the field is the designated field, carrying out data migration on the designated field according to a third mapping rule of the designated field;
and selecting the target data type and the target configuration information which are matched with the specified field in the target database for the specified field by the third mapping rule.
4. The method of data migration of claim 1, further comprising:
generating a first mapping rule and adding the first mapping rule to a mapping rule base;
the generating a first mapping rule comprises:
acquiring source data types of all fields of a source database and candidate data types supported by a target database;
for each field in the data table of the source database, performing the following steps: selecting a target data type which is most matched with the source data type of the field from candidate data types supported by a target database; establishing a mapping relation from a source data type of the source database to a target data type of the target database;
and taking the mapping relations corresponding to the source data types of all the fields in the data table of the source database as a first mapping rule for carrying out data migration from the source data type of the source database to the target data type of the target database.
5. The method of data migration of claim 2, further comprising:
generating a second mapping rule and adding the second mapping rule to a mapping rule base;
the generating a second mapping rule comprises:
acquiring source configuration information of each field of a source database and configuration capacity of a target data type of a target database;
for each field in the data table of the source database, performing the following steps: selecting target configuration information of the target data type which is most matched with the source configuration information of the field according to the configuration capability of the target data type of the target database; establishing a mapping relation from the source configuration information of the field of the source database to the target configuration information of the target data type of the target database;
and taking the mapping relations corresponding to the source configuration information of all fields in the data table of the source database as a second mapping rule for carrying out data migration from the source configuration information of the source database to the target configuration information of the target database.
6. The method of data migration of claim 3, further comprising:
generating a third mapping rule and adding the third mapping rule to a mapping rule base;
the generating the third mapping rule comprises:
selecting a target data type and target configuration information which are most matched with the specified fields from a target database;
establishing a mapping relation from the source data type and the source configuration information of the designated field of the source database to the target data type and the target configuration information of the target database;
and taking the mapping relation corresponding to the specified field as a third mapping rule for performing data migration on the specified field from the source database to the target database.
7. The method for data migration according to any one of claims 1 to 6, wherein the determining whether the source configuration information of each field satisfies the configuration capability of the target data type of the target database includes:
judging whether one or more items of data length, data precision or constraint of each field meet the configuration capability of the target data type of the target database;
wherein the constraint comprises one or more of a primary key constraint with a primary key as a field, a constraint with a unique value as a field, a check constraint with a specified value as a field, a default constraint with a default value as a field, and a foreign key constraint with a foreign key as a field.
8. An apparatus for data migration, comprising:
the acquisition module is configured to acquire a first mapping rule for data migration from a source data type of a source database to a target data type of a target database;
the conversion module is configured to convert the source data type of each field in each data table of the source database into the target data type of the corresponding field in the target database according to the first mapping rule;
the determining module is configured to determine target configuration information of corresponding fields in the target database according to the source configuration information of each field in each data table of the source database and the configuration capability of the target data type of the target database;
and the migration module is configured to generate a target statement for data migration according to the determined target data type and the target configuration information and execute the target statement so as to complete data migration from the source database to the target database.
9. An apparatus for data migration, comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the method of data migration of any of claims 1-7 based on instructions stored in the memory.
10. A system for data migration, comprising:
the source database is used for providing source data types and source configuration information of the fields;
the target database is used for providing candidate data types supported by the target database and corresponding configuration capacity;
and, an apparatus for data migration as claimed in any one of claims 8 to 9.
11. A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of data migration of any of claims 1-7.
CN202110806666.6A 2021-07-16 2021-07-16 Data migration method, device and system Pending CN115617773A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110806666.6A CN115617773A (en) 2021-07-16 2021-07-16 Data migration method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110806666.6A CN115617773A (en) 2021-07-16 2021-07-16 Data migration method, device and system

Publications (1)

Publication Number Publication Date
CN115617773A true CN115617773A (en) 2023-01-17

Family

ID=84855915

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110806666.6A Pending CN115617773A (en) 2021-07-16 2021-07-16 Data migration method, device and system

Country Status (1)

Country Link
CN (1) CN115617773A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116467372A (en) * 2023-02-21 2023-07-21 中国人民解放军海军工程大学 Automatic database conversion method and device, electronic equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116467372A (en) * 2023-02-21 2023-07-21 中国人民解放军海军工程大学 Automatic database conversion method and device, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
US10936625B2 (en) Progressive optimization for implicit cast predicates
CN111767303A (en) Data query method and device, server and readable storage medium
TW200903324A (en) Systems and methods for modeling partitioned tables as logical indexes
CN104965735A (en) Apparatus for generating upgrade SQL script
CN100550020C (en) A kind of method and apparatus that is used to solve the Chinese software issue of supporting multilanguage
CN112948427B (en) Data query method, device, equipment and storage medium
CN106055528A (en) Automatic generation method and apparatus of documents
CN115061721A (en) Report generation method and device, computer equipment and storage medium
US10169414B2 (en) Character matching in text processing
CN113505128A (en) Method, device and equipment for creating data table and storage medium
CN111125229A (en) Data blood margin generation method and device and electronic equipment
CN104714974A (en) Method and device for parsing and reprocessing query statement
EP3474158A1 (en) Method and device for executing distributed computing task
CN111159227B (en) Data query method, device, equipment and storage medium
CN113407565B (en) Cross-database data query method, device and equipment
CN110720097A (en) Functional equivalence of tuples and edges in graph databases
CN115617773A (en) Data migration method, device and system
US20070282804A1 (en) Apparatus and method for extracting database information from a report
CN111984745A (en) Dynamic expansion method, device, equipment and storage medium for database field
CN110502735B (en) Method and system for quickly manufacturing universal process file of electronic equipment
CN104050264A (en) Method and device for generating SQL statement
CN111125216A (en) Method and device for importing data into Phoenix
CN115390936A (en) Unified verification method, device, equipment and storage medium
CN116185389A (en) Code generation method and device, electronic equipment and medium
CN115114297A (en) Data lightweight storage and search method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination