CN114185871A

CN114185871A - Data migration method, data migration device, medium, and electronic device

Info

Publication number: CN114185871A
Application number: CN202111518450.6A
Authority: CN
Inventors: 陈志雄; 刘合曲
Original assignee: China Telecom Corp Ltd
Current assignee: China Telecom Corp Ltd
Priority date: 2021-12-10
Filing date: 2021-12-10
Publication date: 2022-03-15

Abstract

The application provides a data migration method, a data migration device, a data migration medium and electronic equipment, which relate to the technical field of databases and comprise the following steps: acquiring a database table structure of a database to be migrated; carrying out validity check on the database table structure according to a first preset rule to obtain a check result; if the verification result is used for indicating that the database table structure accords with a first preset rule, simulating and migrating the data to be migrated in the database to be migrated to a target database; determining simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database; and if the simulation migration accuracy accords with a second preset rule, migrating the data to be migrated in the database to be migrated to the target database. Therefore, the method can improve the universality of the data migration scheme and the migration accuracy in the formal migration process.

Description

Data migration method, data migration device, medium, and electronic device

Technical Field

The present application relates to the field of database technologies, and in particular, to a data migration method, a data migration apparatus, a computer-readable storage medium, and an electronic device.

Background

Data migration (Data migration) refers to the conversion of Data between storage types, formats, and computer systems. Data migration is often required when a user decides to use a new computing system or use a data management system that is not compatible with the current system. Generally, data migration is achieved manually by modifying configuration parameters of an original database through a professional and linking the original database and a new database. However, this method has high dependency on professionals, and cannot achieve the universality of the data migration scheme.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present application and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.

Disclosure of Invention

The application aims to provide a data migration method, a data migration device, a computer readable storage medium and an electronic device, which can set a first preset rule for limiting a database table structure and a second preset rule for limiting simulation migration accuracy, so that formal data migration can be automatically executed when the simulation migration accuracy is high, thereby reducing the dependency on professionals and improving the universality of a data migration scheme.

Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.

According to an aspect of the present application, there is provided a data migration method, including:

acquiring a database table structure of a database to be migrated;

carrying out validity check on the database table structure according to a first preset rule to obtain a check result;

if the verification result is used for indicating that the database table structure accords with a first preset rule, simulating and migrating the data to be migrated in the database to be migrated to a target database;

determining simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database;

and if the simulation migration accuracy accords with a second preset rule, migrating the data to be migrated in the database to be migrated to the target database.

In an exemplary embodiment of the present application, migrating data to be migrated in a database to be migrated to a target database includes:

and synchronously migrating the data to be migrated in the database to be migrated to the target database through a plurality of processes.

In an exemplary embodiment of the present application, if the check result is used to indicate that the database table structure does not conform to the first preset rule, the method further includes:

decoupling the database table structure;

and displaying the decoupled database table and prompt information for indicating that the database table does not accord with the preset specification on a user interface.

In an exemplary embodiment of the present application, after the data to be migrated in the database to be migrated is migrated to the target database in a simulation manner, the method further includes:

generating a simulation migration report; wherein, the simulation migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration;

the simulated migration report is presented in a user interface.

In an exemplary embodiment of the present application, if the simulation migration accuracy does not meet the second preset rule, the method further includes:

and receiving configuration operation, and configuring any one of fields, the number of migration processes and the migration mode according to the configuration operation.

In an exemplary embodiment of the present application, before obtaining a database table structure of a database to be migrated, the method further includes:

and receiving a link input operation, and responding to the link input operation to establish connection with the database to be migrated according to the input link.

In an exemplary embodiment of the present application, after migrating the data to be migrated in the database to be migrated to the target database, the method further includes:

and if the migration task is not finished, performing incremental supplementary migration on the database to be migrated.

According to an aspect of the present application, there is provided a data migration apparatus including:

the database table structure acquisition unit is used for acquiring a database table structure of the database to be migrated;

the database table structure checking unit is used for checking the legality of the database table structure according to a first preset rule to obtain a checking result;

the data simulation migration unit is used for simulating and migrating the data to be migrated in the database to be migrated to the target database when the verification result is used for indicating that the database table structure conforms to the first preset rule;

the simulation migration accuracy determining unit is used for determining simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database;

and the data formal migration unit is used for migrating the data to be migrated in the database to be migrated to the target database when the simulation migration accuracy conforms to a second preset rule.

In an exemplary embodiment of the present application, migrating data to be migrated in a database to be migrated to a target database by a data formal migration unit includes:

In an exemplary embodiment of the present application, the apparatus further includes:

the structure unit is used for decoupling the database table structure when the verification result is used for indicating that the database table structure does not accord with the first preset rule;

and the front-end display unit is used for displaying the decoupled database table on the user interface and indicating prompt information that the database table does not accord with the preset specification.

the simulation migration report generation unit is used for generating a simulation migration report after the data simulation migration unit simulates and migrates the data to be migrated in the database to be migrated to the target database; wherein, the simulation migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration;

and the front-end display unit is also used for displaying the simulation migration report in the user interface.

and the configuration unit is used for configuring any one of the fields, the migration process quantity and the migration mode according to the configuration operation when the simulation migration accuracy does not accord with the second preset rule and the configuration operation is received.

and the database connection unit is used for receiving a link input operation before the database table structure acquisition unit acquires the database table structure of the database to be migrated, and responding to the link input operation to establish connection with the database to be migrated according to the input link.

and the incremental migration unit is used for performing incremental supplementary migration on the database to be migrated if the migration task is not finished after the data formal migration unit migrates the data to be migrated in the database to be migrated to the target database.

According to an aspect of the present application, there is provided an electronic device including: a processor; and a memory for storing executable instructions for the processor; wherein the processor is configured to perform the method of any of the above via execution of the executable instructions.

According to an aspect of the application, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the above.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations described above.

The exemplary embodiments of the present application may have some or all of the following advantages:

in the data migration method provided by an example embodiment of the present application, a database table structure of a database to be migrated may be obtained; carrying out validity check on the database table structure according to a first preset rule to obtain a check result; if the verification result is used for indicating that the database table structure accords with a first preset rule, simulating and migrating the data to be migrated in the database to be migrated to a target database; determining simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database; and if the simulation migration accuracy accords with a second preset rule, migrating the data to be migrated in the database to be migrated to the target database. According to the above scheme description, on one hand, a first preset rule for limiting the database table structure and a second preset rule for limiting the simulation migration accuracy can be set, so that formal data migration can be automatically executed when the simulation migration accuracy is high, the dependency on professionals is reduced, and the universality of a data migration scheme is improved. On the other hand, the migration accuracy in the formal migration process can be improved based on the limitation of the second preset rule on the simulation migration accuracy.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.

Fig. 1 is a schematic diagram illustrating an exemplary system architecture to which a data migration method and a data migration apparatus according to an embodiment of the present application may be applied;

FIG. 2 illustrates a schematic structural diagram of a computer system suitable for use in implementing an electronic device of an embodiment of the present application;

FIG. 3 schematically illustrates a flow diagram of a method of data migration according to one embodiment of the present application;

FIG. 4 schematically shows an architectural diagram for implementing a data migration method according to an embodiment of the present application;

FIG. 5 schematically illustrates a model structure for performing a data migration method according to one embodiment of the present application;

FIG. 6 schematically shows a program catalog diagram for implementing a data migration method according to one embodiment of the present application;

FIG. 7 schematically illustrates a flow diagram of a method of data migration according to one embodiment of the present application;

fig. 8 schematically shows a block diagram of a data migration apparatus according to an embodiment of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present application.

Furthermore, the drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

Fig. 1 is a schematic diagram illustrating a system architecture of an exemplary application environment to which a data migration method and a data migration apparatus according to an embodiment of the present application may be applied.

As shown in fig. 1, the system architecture 100 may include one or more of

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few. The

terminal devices

101, 102, 103 may be various electronic devices having a display screen, including but not limited to desktop computers, portable computers, smart phones, tablet computers, and the like. It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. For example, server 105 may be a server cluster comprised of multiple servers, or the like.

The data migration method provided by the embodiment of the present application is generally executed by the server 105, and accordingly, the data migration apparatus is generally disposed in the server 105. However, it is easily understood by those skilled in the art that the data migration method provided in the embodiment of the present application may also be executed by the

terminal device

101, 102, or 103, and accordingly, the data migration apparatus may also be disposed in the

terminal device

101, 102, or 103, which is not particularly limited in this exemplary embodiment. For example, in one exemplary embodiment, server 105 may obtain a database table structure of a database to be migrated; carrying out validity check on the database table structure according to a first preset rule to obtain a check result; if the verification result is used for indicating that the database table structure accords with a first preset rule, simulating and migrating the data to be migrated in the database to be migrated to a target database; determining simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database; and if the simulation migration accuracy accords with a second preset rule, migrating the data to be migrated in the database to be migrated to the target database.

FIG. 2 illustrates a schematic structural diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

It should be noted that the computer system 200 of the electronic device shown in fig. 2 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 2, the computer system 200 includes a Central Processing Unit (CPU)201 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)202 or a program loaded from a storage section 208 into a Random Access Memory (RAM) 203. In the RAM 203, various programs and data necessary for system operation are also stored. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204. An input/output (I/O) interface 205 is also connected to bus 204.

The following components are connected to the I/O interface 205: an input portion 206 including a keyboard, a mouse, and the like; an output section 207 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 208 including a hard disk and the like; and a communication section 209 including a network interface card such as a LAN card, a modem, or the like. The communication section 209 performs communication processing via a network such as the internet. A drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is mounted on the drive 210 as necessary, so that a computer program read out therefrom is installed into the storage section 208 as necessary.

In particular, according to embodiments of the present application, the processes described below with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 209 and/or installed from the removable medium 211. The computer program, when executed by a Central Processing Unit (CPU)201, performs various functions defined in the methods and apparatus of the present application.

The example embodiment provides a data migration method. The data migration method may be applied to the server 105, and may also be applied to one or more of the

terminal devices

101, 102, and 103, which is not particularly limited in this exemplary embodiment. Referring to fig. 3, the data migration method may include the following steps S310 to S350.

Step S310: and acquiring a database table structure of the database to be migrated.

Step S320: and carrying out validity check on the database table structure according to a first preset rule to obtain a check result.

Step S330: and if the verification result is used for indicating that the database table structure accords with a first preset rule, simulating and migrating the data to be migrated in the database to be migrated to the target database.

Step S340: and determining the simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database.

Step S350: and if the simulation migration accuracy accords with a second preset rule, migrating the data to be migrated in the database to be migrated to the target database.

The technical solutions of steps S310 to S350 in the embodiment of the present application may be developed based on python programming language.

By implementing the method shown in fig. 3, a first preset rule for defining the database table structure and a second preset rule for defining the simulation migration accuracy can be set, so that formal data migration can be automatically executed when the simulation migration accuracy is high, the dependency on professionals is reduced, and the universality of a data migration scheme is improved. In addition, the migration accuracy in the formal migration process can be improved based on the limitation of the second preset rule on the simulation migration accuracy.

The above steps of the present exemplary embodiment will be described in more detail below.

In step S310, a database table structure of the database to be migrated is obtained.

Specifically, the database to be migrated and the target database may be any one of MySQL, SQLServer, Oracle, DB2, SyBase, Informix, PostgreSQL, and the like, and the embodiment of the present application is not limited thereto.

Wherein, the database table structure can be represented as follows:

referring to fig. 4, fig. 4 schematically shows an architecture diagram of a method for implementing data migration according to an embodiment of the present application. As shown in fig. 4, the architecture for implementing the data migration method includes: a front-end user interface 410, a data processing platform 420, a data collection module 430, and a database engine 440.

The front-end user interface 410 may be constructed based on HTML 5411, TomCat 412, Javascript 413, J2ee 414, or boottrap 415, among others. The data processing platform 420 may include a table structure analysis module 421, a data analysis module 422, and a data migration module 423; the table structure analysis module 421 may be configured to perform validity check on a database table structure according to a first preset rule, the data analysis module 422 is configured to analyze data to be migrated in the database to be migrated, and the data migration module 423 is configured to simulate migration of the data to be migrated in the database to be migrated to a target database and migrate the data to be migrated in the database to be migrated to the target database. The data acquisition module 430 includes a data preprocessing function 431, a data table structure acquisition function 432, and a data acquisition function 433; the data preprocessing function 431 is used for unifying data formats, the data table structure acquisition function 432 is used for acquiring a database table structure of the database to be migrated, and the data acquisition function 433 is used for acquiring data to be migrated in the database to be migrated. The database engine 440 includes a to-be-migrated database 441 and a target database 442; the database to be migrated 441 may include Oracle4411, Informix4412, SQL Server4413, DB24414 and MySQL4415, and the target database 442 may include TeleDB4421, TelPG4422 and Udal 4423.

As an alternative embodiment, before obtaining the database table structure of the database to be migrated, the method further includes: and receiving a link input operation, and responding to the link input operation to establish connection with the database to be migrated according to the input link.

Specifically, the link input operation may be a click operation, a paste operation, a drag operation, and the like, and the embodiment of the present application is not limited thereto, and the link input operation is used to input a database link in a link input area of the front-end user interface. Establishing connection with the database to be migrated according to the input link, wherein the establishing connection comprises the following steps: sending a connection request to the input link, and if receiving connection permission feedback, establishing connection with the database to be migrated; wherein the input link corresponds to the database to be migrated.

In addition, acquiring a database table structure of the database to be migrated, including: and reading the database table structure and the data to be migrated from the database to be migrated, and uploading the database table structure and the data to be migrated to the synchronous system database so as to be called from the synchronous system database at any time.

Therefore, by implementing the optional embodiment, the connection between the target database and each database can be established according to the link, so that the data migration from each database to the target database is realized, and the data migration efficiency is improved.

In step S320, a validity check is performed on the database table structure according to a first preset rule to obtain a check result.

Specifically, the first preset rule may be formed by configuration parameters, and is used to determine whether the database table structure meets the configuration parameters, where the configuration parameters may be represented by numerical values, character strings, symbols, and the like, and the embodiment of the present application is not limited to specific configuration parameters in the first preset rule.

In step S330, if the check result is used to indicate that the database table structure meets the first preset rule, the data to be migrated in the database to be migrated is migrated to the target database in a simulation manner.

Specifically, the target database may be a self-developed database, such as TeleDB, TelPG, Udal, and the like. The method for simulating and migrating the data to be migrated in the database to be migrated to the target database comprises the following steps: and generating a control instruction for constructing the database table in the target database, wherein the control instruction can be represented by a DLL statement, and further the database table can be constructed in the target database through the control instruction.

In addition, if the check result is used to indicate that the database table structure does not conform to the first preset rule, the method may further include: and generating a modification suggestion aiming at the database table structure, and outputting the modification suggestion to a user interface so as to facilitate relevant personnel to modify the configuration parameters of the database table structure by referring to the modification suggestion. When the modification completion is detected, step S320 is performed.

As an alternative embodiment, if the verification result is used to indicate that the database table structure does not conform to the first preset rule, the method further includes: decoupling the database table structure; and displaying the decoupled database table and prompt information for indicating that the database table does not accord with the preset specification on a user interface.

Specifically, after the user interface displays the decoupled database table and the prompt information for indicating that the database table does not conform to the preset specification, the method may further include: receiving configuration information input by a user in a user interface, reconfiguring the database table structure according to the configuration information, and carrying out validity check on the reconfigured database table structure according to a first preset rule.

Therefore, by implementing the optional embodiment, the visualization processing of the related information of the database can be realized, so that a user can know the legality checking result of the database table through the user interface, and can also directly input corresponding configuration information through the user interface, thereby improving the universality, convenience and interactivity of the data migration scheme.

As an optional embodiment, after the data to be migrated in the database to be migrated is migrated to the target database in a simulation manner, the method further includes: generating a simulation migration report; wherein, the simulation migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration; the simulated migration report is presented in a user interface.

In particular, the simulated migration report may also be used to indicate the completion (e.g., 90%) of the data migration. In addition, the simulation migration report may further include a generated parameter adjustment suggestion for reference by the relevant person, where the parameter adjustment suggestion is used to assist the relevant person in reconfiguring parameters, so as to improve the data volume of correct migration in the simulation migration process.

Therefore, by implementing the optional embodiment, the configuration parameters of the migration process can be adjusted continuously by the user through the simulation data synchronization and the display of the simulation migration report, so that the accuracy and the completion degree of the data migration are improved.

In step S340, a simulation migration accuracy is determined according to the data to be migrated in the database to be migrated and the data to be migrated in the target database.

Specifically, the type of the database to be migrated and the type of the target database may be the same or different. The simulation migration accuracy can be represented by a numerical value/percentage, and the simulation migration accuracy is used for representing the proportion of the data to be migrated, which is correctly migrated from the database to be migrated to the target database, in all the data to be migrated.

In addition, if the simulation migration accuracy does not meet the second preset rule, the method may further include: and outputting prompt information for prompting a user to reconfigure the parameters, and executing the step of simulating and migrating the data to be migrated in the database to be migrated to the target database after the configuration is detected to be completed. Therefore, the data migration problem can be conveniently processed by a user in an iterative mode through multiple times of simulation migration.

In step S350, if the simulation migration accuracy meets the second preset rule, the data to be migrated in the database to be migrated is migrated to the target database.

In particular, the second preset rule may include an accuracy threshold (e.g., 80%), and the second preset rule is used to define a condition for complying with formal data migration. The number of the target databases may be one or more, and if the number of the target databases is multiple, the manner of migrating the data to be migrated in the database to be migrated to the target database may specifically be: migrating the data to be migrated in the database to be migrated to each target database; each target database may be deployed in node servers in different areas. Therefore, data migration from one database to be migrated to a plurality of target databases is achieved, the target databases can be the same type databases in different areas and can be synchronous databases and used for storing the same data, the problem that data cannot be called due to the fact that a single database is in a problem is avoided, and the usability of the data is guaranteed.

In addition, after the data to be migrated in the database to be migrated is migrated to the target database, the method may further include: determining migration efficiency (e.g., 22.6M/s) in a migration process, formal migration time (e.g., 8.1 minutes), migration work development time (e.g., 1 month 10 to 1 month 11 in 2021), abnormal data volume (e.g., 12131), actual migration data volume (e.g., 2827832), number of abnormal tables (e.g., 13), total number (e.g., 11G), total migration entries (e.g., 2839963), number of total migration data tables (e.g., 273), user name and other migration information, displaying migration information on a user interface, and uploading the migration information to a server, so that the server stores the migration information generated by the current migration, and can facilitate subsequent invocation.

In addition, after the data to be migrated in the database to be migrated is migrated to the target database, the method further includes: generating a formal migration report; the formal migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration; the simulated migration report is presented in a user interface. Further, the method may further include: and performing data analysis on the normal migration report and the simulated migration report to obtain report analysis for representing report difference and displaying the report analysis, so that a user can know the difference between the simulated migration result and the formal migration result according to the report analysis. Optionally, the formal migration report and the simulated migration report may include at least one of the migration information described above.

As an optional embodiment, migrating the data to be migrated in the database to be migrated to the target database includes: and synchronously migrating the data to be migrated in the database to be migrated to the target database through a plurality of processes.

Specifically, a process is a process of one-time dynamic execution of a program with an independent function on a data set, is an independent unit for resource allocation and scheduling by an operating system, and is also a carrier for running an application program. The process generally consists of three parts, namely a program, a data set and a process control block; the program is used for describing functions to be completed by the process and is an instruction set for controlling the process to execute; a data set is data and a workspace required by a program when executing; the program control block contains the description information and control information of the process and is the only mark of the process existence.

The method for synchronously migrating the data to be migrated in the database to be migrated to the target database through a plurality of processes comprises the following steps: and synchronously performing data conversion on the data to be migrated in the database to be migrated in the memory through a plurality of processes, and migrating the result after the data conversion to the target database. Therefore, the hardware capability and the network capability can be exerted to the maximum extent, and the data migration efficiency is improved.

Therefore, by implementing the optional embodiment, synchronous migration can be performed on the data to be migrated based on a plurality of processes, so that migration efficiency can be improved.

As an alternative embodiment, if the simulation migration accuracy does not meet the second preset rule, the method further includes: and receiving configuration operation, and configuring any one of fields, the number of migration processes and the migration mode according to the configuration operation.

Specifically, a configuration operation is received, comprising: a configuration operation is received in a user interface.

Therefore, the implementation of the optional embodiment can facilitate the user to execute corresponding configuration operation according to the simulation migration accuracy, thereby improving the accuracy and the completion of formal data migration.

As an optional embodiment, after migrating the data to be migrated in the database to be migrated to the target database, the method further includes: and if the migration task is not finished, performing incremental supplementary migration on the database to be migrated.

Specifically, performing incremental supplementary migration on the database to be migrated includes: and when detecting that newly added data to be migrated appear in the database to be migrated, performing incremental supplementary migration on the database to be migrated. The timestamp of the newly added data to be migrated and the timestamp of the original data to be migrated can belong to different time periods.

Therefore, the implementation of the alternative embodiment can improve the automation degree of the data migration scheme.

Referring to fig. 5, fig. 5 schematically illustrates a model structure for performing a data migration method according to an embodiment of the present application. As shown in fig. 5, the model for performing the data migration method may include: the database synchronization system comprises a synchronization database module 510, a database table information module 520, a database table structure checking module 530, a database table field information module 540, a data synchronization configuration module 550, a data synchronization log module 560 and an abnormal data table module 570.

The synchronization database module 510 is configured to record information of the database to be migrated and a data migration progress (e.g., a synchronization state), and may specifically include: database name, database type, user name (i.e., the name of the user who has a data migration requirement), password, database link name, synchronization status. The database table information module 520 is configured to record related information of a database table, and may specifically include: database name, database table ID, owner name (i.e., the owner of the database table), database table size, number of rows stored, synchronization status. The database table structure check module 530 can perform a validity check of a database table with the following information: database name, database table ID, verification result, verification time and database table modification suggestion. The database table field information module 540 is configured to record field information related to a database table, and specifically may include: database name, database table name, owner name, field type, and field length. The data synchronization configuration module 550 may perform a data migration scheme using the following information: database name, data synchronization batch ID, database table ID, target database, verification time, source data entry, synchronization data entry, read data entry, and synchronization state update time. The data synchronization log module 560 is configured to store a log recording a data migration process, and specifically may include: database name, data synchronization batch ID, start batch number, start time, end time, synchronization data entry, error reporting information, and modulus value. The abnormal data table module 570 is configured to record relevant information of an abnormal data table in the data migration process, and specifically may include: data synchronization batch ID, database table name, database table ID, field 1 value, field 2 value and field 3 value; the field 1 value, the field 2 value, and the field 3 value may be used to record values corresponding to different fields.

Referring to fig. 6, fig. 6 schematically illustrates a program directory diagram for implementing a data migration method according to an embodiment of the present application. As shown in fig. 6, a program directory for implementing a data migration method may be represented by a master directory (SyncData)600, wherein the master directory (SyncData)600 may include: a program directory (PROCESSOR)610, a configuration directory (CONF)620, a REPORT directory (REPORT)630, a LOG directory (LOG)640, an interface directory (WebContent)650, a main program (main _ PROCESSOR. py)660, a start program (start _ sync. sh)670, and a stop program (stop _ sync. sh) 680.

Specifically, the program catalog (PROCESSOR)610 may include: the database engine program (database _ engine. py)611, the data synchronization program (sync _ table _ normal. py)612, the data structure analysis program (analysis _ table _ info. py)613, the synchronization data analysis program (stat _ sync. py)614, and the database engine program (database _ engine. py)611 includes an oracle interface class (Ora _ engine)6111, an informix interface class (inf _ engine)6112, a DB2 interface class (DB2_ engine)6113, an sql interface class (MYSQL _ engine)6114, an sqverer interface class (Sqls _ engine)6115, an oracle interface class (telestrip _ engine)6116, a telepgen interface class (telepgen _ engine)6117, and an Udal interface class (ueda _ engine _ 18).

The configuration directory (CONF)620 may include: data base json 621 and program base json 622. The REPORT catalog (REPORT)630 may include: table institution verification report 631, simulation data synchronization report 632, and formal data synchronization report 633. The LOG directory (LOG)640 may include: a program run log (run _ yyyymmdlog) 641 and a run exception log (error _ yyyymmdd.log) 642. The interface directory (WebContent)650 may include: css651, img652, function interface directory (link)653, login interface (index. jsp)654, and main interface (main _ index. jsp) 655; wherein the css651 may be specifically a framework introduction package 6511, the img652 may be specifically a picture 6521, and the function interface directory (link)653 may include a configuration interface (config synctable. jsp)6531, an analysis interface (analytics sync. jsp)6532, and a synchronization health interface (moniertsync. jsp) 6533.

Wherein, the program directory (PROCESSOR)610 executes the steps S310 to S350. A REPORT catalog (REPORT)630 is used to generate data migration REPORTs. The configuration directory (CONF)620 is used to implement parameter configuration during data migration. A LOG directory (LOG)640 is used to record the data migration process. The interface directory (WebContent)650 is used to visually present the data migration results in the user interface.

Referring to FIG. 7, FIG. 7 schematically illustrates a flow diagram of a method of data migration according to an embodiment of the present application. As shown in fig. 7, the data migration method includes: step S700 to step S790.

Step S700: and receiving a link input operation, and responding to the link input operation to establish connection with the database to be migrated according to the input link.

Step S710: and acquiring a database table structure of the database to be migrated, and carrying out validity check on the database table structure according to a first preset rule to obtain a check result. If the check result is used to indicate that the database table structure conforms to the first predetermined rule, step S820 is executed. If the check result is used to indicate that the database table structure does not conform to the first preset rule, step S830 is performed.

Step S720: and simulating and migrating the data to be migrated in the database to be migrated to the target database.

Step S730: and decoupling the database table structure, displaying the decoupled database table and prompt information used for indicating that the database table is not in accordance with the preset specification on a user interface, and receiving parameter adjustment operation. Then, step S710 is performed.

Step S740: and generating a simulated migration report, wherein the simulated migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration, and further displaying the simulated migration report in a user interface.

Step S750: and determining the simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database. If the simulation migration accuracy meets the second predetermined rule, step S760 is executed. If the simulation migration accuracy does not meet the second preset rule, step S770 is executed.

Step S760: and synchronously migrating the data to be migrated in the database to be migrated to the target database through a plurality of processes.

Step S770: and receiving configuration operation, and configuring any one of fields, the number of migration processes and the migration mode according to the configuration operation. Then, step S750 is executed.

Step S780: and generating a formal migration report, wherein the simulated migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration, and further displaying the formal migration report in a user interface.

Step S790: and if the migration task is not finished, performing incremental supplementary migration on the database to be migrated.

It should be noted that steps S700 to S790 correspond to the steps and embodiments shown in fig. 3, and for the specific implementation of steps S700 to S790, please refer to the steps and embodiments shown in fig. 3, which are not described herein again.

It can be seen that, by implementing the method shown in fig. 7, a first preset rule for defining a database table structure and a second preset rule for defining simulation migration accuracy can be set, so that formal data migration can be automatically executed when the simulation migration accuracy is high, thereby reducing dependency on professionals and improving the universality of a data migration scheme. In addition, the migration accuracy in the formal migration process can be improved based on the limitation of the second preset rule on the simulation migration accuracy.

Referring to fig. 8, fig. 8 is a block diagram schematically illustrating a data migration apparatus according to an embodiment of the present application. As shown in fig. 8, the data migration apparatus 800 includes:

a database table structure obtaining unit 801, configured to obtain a database table structure of a database to be migrated;

a database table structure checking unit 802, configured to perform validity checking on a database table structure according to a first preset rule to obtain a checking result;

the data simulation migration unit 803 is configured to, when the verification result is used to indicate that the database table structure conforms to the first preset rule, simulate migration of the to-be-migrated data in the to-be-migrated database to the target database;

a simulation migration accuracy determining unit 804, configured to determine simulation migration accuracy according to the data to be migrated in the database to be migrated and the data to be migrated in the target database;

and the data formal migration unit 805 is configured to migrate the data to be migrated in the database to be migrated to the target database when the simulation migration accuracy meets a second preset rule.

It can be seen that, by implementing the apparatus shown in fig. 8, a first preset rule for defining a database table structure and a second preset rule for defining simulation migration accuracy may be set, so that formal data migration may be automatically performed when the simulation migration accuracy is high, thereby reducing dependency on professionals and improving the universality of a data migration scheme. In addition, the migration accuracy in the formal migration process can be improved based on the limitation of the second preset rule on the simulation migration accuracy.

In an exemplary embodiment of the present application, the data formal migration unit 805 migrates the data to be migrated in the database to be migrated to the target database, including:

a structural unit (not shown) for decoupling the database table structure when the check result is used for indicating that the database table structure does not conform to the first preset rule;

and the front-end display unit (not shown) is used for displaying the decoupled database table and prompt information for indicating that the database table does not accord with the preset specification on the user interface.

a simulation migration report generation unit (not shown) configured to generate a simulation migration report after the data simulation migration unit 803 simulates and migrates the data to be migrated in the database to be migrated to the target database; wherein, the simulation migration report is used for representing the data volume of the correct migration and the data volume of the abnormal migration;

and a configuration unit (not shown) configured to configure any one of the field, the number of migration processes, and the migration mode according to the configuration operation when the simulation migration accuracy does not meet the second preset rule and the configuration operation is received.

a database connection unit (not shown) configured to receive a link input operation before the database table structure acquisition unit 801 acquires the database table structure of the database to be migrated, and establish a connection with the database to be migrated according to the input link in response to the link input operation.

an incremental migration unit (not shown) configured to, after the data formal migration unit 805 migrates the data to be migrated in the database to be migrated to the target database, perform incremental supplementary migration on the database to be migrated if it is detected that the migration task is not finished.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

For details that are not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the data migration method described above for the details that are not disclosed in the embodiments of the apparatus of the present application.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method described in the above embodiments.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method of data migration, comprising:

acquiring a database table structure of a database to be migrated;

if the verification result is used for indicating that the database table structure accords with the first preset rule, simulating and migrating the data to be migrated in the database to be migrated to a target database;

2. The method of claim 1, wherein migrating the data to be migrated in the database to be migrated to the target database comprises:

3. The method according to claim 1, wherein if the check result indicates that the database table structure does not conform to the first predetermined rule, the method further comprises:

decoupling the database table structure;

and displaying the decoupled database table and prompt information for indicating that the database table is not in accordance with the preset specification on a user interface.

4. The method according to claim 1, wherein after the data to be migrated in the database to be migrated is simulated and migrated into the target database, the method further comprises:

generating a simulation migration report; wherein the simulated migration report is used for representing the data volume of correct migration and the data volume of abnormal migration;

displaying the simulated migration report in a user interface.

5. The method of claim 1, wherein if the simulation migration accuracy does not meet the second predetermined rule, the method further comprises:

6. The method of claim 1, wherein prior to obtaining the database table structure of the database to be migrated, the method further comprises:

7. The method of claim 1, wherein after migrating the data to be migrated in the database to be migrated into the target database, the method further comprises:

8. A data migration apparatus, comprising:

the data simulation migration unit is used for simulating and migrating the data to be migrated in the database to be migrated to a target database when the verification result is used for indicating that the database table structure conforms to the first preset rule;

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 7.

10. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any of claims 1-7 via execution of the executable instructions.