CN115080540A - Data migration method and device, computer equipment and storage medium - Google Patents

Abstract

The application relates to a data migration method, a data migration device, computer equipment and a storage medium, which are suitable for various systems needing data migration, such as an Enterprise Resource Planning (ERP) system. The method comprises the following steps: performing data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly connecting different external systems; migrating the data conforming to the intermediate table structure to the universal intermediate table; and performing adaptation processing corresponding to a target system on the data in the universal intermediate table, and writing the data after the adaptation processing into a target data table in the target system. By adopting the method, the data migration efficiency can be improved.

Description

Data migration method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data migration method and apparatus, a computer device, and a storage medium.

Background

With the development of computer technology, various systems come into endless, and are continuously updated and iterated, for example: systems such as Human Resources Management Information System (HRMIS) and Customer Relationship Management System (CRM) used by enterprises have been developed by many manufacturers. With the development of enterprises, enterprises sometimes need to replace new systems to meet their business and management needs, and thus how to migrate data of an original system to a target system becomes a very important issue.

In the conventional method, data is migrated from an original system to a target system through extraction-Transform-Load (ETL) tool through extraction-Transform-Load (e.g., Extract-Transform-Load) and loading. However, this method can only be used for data migration between relatively simple isomorphic systems, and in practical applications, the difference of data structures between systems developed by different manufacturers is large, and it is difficult to implement migration through ETL tools, but a large amount of SQL (Structured Query Language) scripts need to be manually written, which is poor in maintainability, very time-consuming and labor-consuming, and thus, is very low in efficiency.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data migration method, apparatus, computer device, computer readable storage medium and computer program product, which can improve efficiency.

In a first aspect, the present application provides a data migration method. The method comprises the following steps:

performing data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly connecting different external systems;

migrating the data conforming to the intermediate table structure to the universal intermediate table;

and performing adaptation processing corresponding to a target system on the data in the universal intermediate table, and writing the data after the adaptation processing into a target data table in the target system.

In a second aspect, the present application further provides a data migration apparatus. The device comprises:

the data mapping module is used for performing data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly butting different external systems;

the data transmission module is used for migrating the data conforming to the intermediate table structure to the universal intermediate table;

and the data writing module is used for carrying out adaptation processing corresponding to a target system on the data in the universal intermediate table and writing the data after the adaptation processing into a target data table in the target system.

In one embodiment, the apparatus further comprises:

the data acquisition module is used for determining the configured extension migration field item; the extended migration field item is an extended field item used for indicating data to be migrated; expanding the field items in the universal intermediate table according to the expanded migration field items; acquiring corresponding data in the original system according to a preset migration field item and the extended migration field item to obtain the data to be migrated; the preset migration field item is a preset field item used for indicating data to be migrated.

In one embodiment, the data writing module is further configured to add a tag field to the data after the adaptation process; and writing the data which is subjected to the adaptation processing and carries the mark field into a target data table in the target system.

In one embodiment, the data after the adaptation process comprises target data adapted to a base value in the target system; the data writing module is also used for carrying out basic verification on the data in the universal intermediate table; and converting the data in the universal intermediate table after the basic verification into target data matched with the basic data value in the target system.

In one embodiment, the data writing module is further configured to perform at least one of the following steps: carrying out uniqueness check on the data in the universal intermediate table; carrying out field value deletion verification on mandatory field items in the universal intermediate table; and matching and checking the field values in the universal intermediate table and the corresponding basic data values in the target system.

In one embodiment, the structure mapping transformation process is performed by an integrated platform; preset mapping schemes aiming at different external systems are integrated in the integrated platform; the data mapping module is also used for determining a preset mapping scheme corresponding to the original system from preset mapping schemes integrated in the integration platform and aiming at different external systems; the data transmission module is also used for migrating the data which are in accordance with the intermediate table structure to the universal intermediate table through the integration platform.

In one embodiment, the data writing module is further configured to perform, by a data migration tool integrated in a target system, adaptation processing corresponding to the target system on data in the universal intermediate table; and calling a preset interface in the target system through the data migration tool, and writing the data after the adaptation processing into a target data table in the target system.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to execute the steps of the data migration method according to the embodiments of the present application.

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer readable storage medium has a computer program stored thereon, and when executed by a processor, the computer program causes the processor to execute the steps in the data migration method according to the embodiments of the present application.

In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the computer program causes the processor to execute the steps in the data migration method according to the embodiments of the present application.

According to the data migration method, the data structure mapping conversion processing is carried out on the data to be migrated in the original system according to the preset mapping scheme corresponding to the original system, so as to obtain the data conforming to the intermediate table structure, the intermediate table structure is the preset data structure required by the universal intermediate table for uniformly connecting different external systems, so that the data of different original systems can be uniformly mapped into the data conforming to the intermediate table structure without writing complex scripts by a user, then the data conforming to the intermediate table structure is migrated into the universal intermediate table, the data in the universal intermediate table is subjected to the adaptation processing corresponding to the target system, the data subjected to the adaptation processing is written into the target data table in the target system, and the high-efficiency data migration between the original system and the target system with different data structures is realized, the data migration efficiency is improved.

Drawings

FIG. 1 is a diagram of an application environment for a data migration method in one embodiment;

FIG. 2 is a diagram of an application environment of a data migration method in another embodiment;

FIG. 3 is a flow diagram that illustrates a method for data migration in one embodiment;

FIG. 4 is a schematic diagram illustrating an overall flowchart of a data migration method according to an embodiment;

FIG. 5 is a diagram that illustrates the interaction between the universal intermediate table and the target data table, in one embodiment;

FIG. 6 is a diagram that illustrates data migration according to a template class, in one embodiment;

FIG. 7 is a block diagram of a data migration apparatus in one embodiment;

FIG. 8 is a block diagram showing the construction of a data migration apparatus according to another embodiment;

FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The data migration method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. The server 102 of the original system communicates with the data migration server 104 through a network, and the data migration server 104 communicates with the server 106 of the target system through a network. The data storage system may store data that the server 102 of the original system, the data migration server 104, and the server 106 of the target system need to process. The data storage system may be integrated on the server 102 of the original system, the data migration server 104 and the server 106 of the target system respectively, or may be placed on a cloud or other network server.

Specifically, the data migration server 104 may obtain data to be migrated from the server 102 of the original system, and the data migration server 104 may perform data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure, then migrate the data conforming to the intermediate table structure into the general intermediate table, perform adaptation processing corresponding to the target system on the data in the general intermediate table, and write the data after the adaptation processing into the target data table provided by the server 106 of the target system.

The server 102 of the original system, the data migration server 104 and the server 106 of the target system may be implemented by independent servers or a server cluster formed by a plurality of servers.

In one embodiment, an integration platform and data migration tools may be hosted on the data migration server 104. The integration platform can perform data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure, and then migrate the data conforming to the intermediate table structure to a general intermediate table in the data migration tool. The data migration tool may perform adaptation processing corresponding to the target system on the data in the general intermediate table, and write the data after the adaptation processing into a target data table provided by the server 106 of the target system.

In another embodiment, the functions of the data migration server 104 may be implemented by different servers, such as: the integration platform and the data migration tool can be respectively carried on different servers. For example: the integrated platform may be mounted on an integrated platform server, and the data migration tool may be mounted on a data migration tool server.

In other embodiments, the data migration server 104 and the server 106 of the target system may be implemented by the same server.

In one embodiment, as shown in FIG. 2, the terminal 108 and the data migration server 104 communicate over a network. The user may configure the data migration task through the terminal 108, and the terminal 108 may send the data migration task to the data migration server 104 to notify the data migration server 104 to start data migration. The terminal 108 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart car-mounted devices, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like.

In one embodiment, as shown in fig. 3, a data migration method is provided, which is described by taking the method as an example applied to the data migration server 104 in fig. 1, and includes the following steps:

step 302, performing data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly interfacing different external systems.

Wherein, the original system is the system originally used by the user. Data in the original system needs to be migrated to the target system. The preset mapping scheme is a scheme which is respectively preset for different original systems and is used for performing data structure mapping conversion processing on the data to be migrated in the original systems. The data to be migrated is data that needs to be migrated to the target system. And the data structure mapping conversion processing is to convert the data structure of the data to be migrated from the data structure mapping of the original system into an intermediate table structure. And the universal intermediate table is used for receiving data of an external system.

The type of the system involved in the embodiment of the present application is not limited, and may be, for example, an Enterprise Resource Planning (ERP) system, where the ERP system includes, but is not limited to, a production manufacturing management system, a supply chain system, a human Resource system, a financial system, a tax system, and an auditing system.

In one embodiment, the original system and the target system may be the same function system, such as: both the original System and the target System are Human Resources Management Information Systems (HRMIS), such as: both the original system and the target system are Customer Relationship Management systems (CRM).

In one embodiment, the data structures of the data in the original system and the target system may be the same or different.

In one embodiment, the data to be migrated may be data corresponding to a field entry that needs to be migrated in the original system. In one embodiment, the data to be migrated may be data in the original system except at least one of data corresponding to a field item specific to the original system, data corresponding to a field item at a system level, data corresponding to a field item set by a user to be unnecessary to be migrated, and the like.

In one embodiment, the data structure mapping transformation process may include at least one of: directly mapping a data table in an original system into a universal intermediate table; mapping a plurality of data tables in an original system into a universal intermediate table; mapping rows in an original system data table into columns in a universal intermediate table; converting the column mapping in the original system data table into rows in a universal intermediate table; and converting the enumerated value of the field in the original system data table into the enumerated value of the field in the target system data table.

In one embodiment, an integration platform may be installed on the data migration server, and a preset mapping scheme may be integrated in the integration platform. The data migration server can perform data structure mapping conversion processing on the data to be migrated in the original system through the integrated platform according to a preset mapping scheme corresponding to the original system to obtain data conforming to the intermediate table structure. In one embodiment, different preset mapping schemes for different original systems may be integrated into the integration platform. The integrated platform is a software platform supporting application development and system integrated operation in a complex information environment.

As shown in step 1 "extraction" in fig. 4, the integration platform may perform data structure mapping conversion processing on the data tables of different data sources (i.e., data source a, data source B, and data source C) to obtain data conforming to the intermediate table structure.

And step 304, migrating the data conforming to the intermediate table structure to the universal intermediate table.

In one embodiment, the data migration server may migrate data conforming to the intermediate table structure into the universal intermediate table through the integration platform.

In one embodiment, a common intermediate table may be provided in the data migration tool.

In one embodiment, the data migration tool may be integrated into the target system.

In one embodiment, the data migration server may transmit the data conforming to the intermediate table structure to the data migration tool, and the data migration tool may write the transmitted data into the general intermediate table, so as to implement migration of the data conforming to the intermediate table structure into the general intermediate table.

In one embodiment, the data migration server may define that only incremental data migration is allowed. It can be understood that through the setting of incremental data migration, a user can add data to be migrated at any time in the migration process, so that the flexibility is improved, and the safety and reliability of the data can be improved.

In one embodiment, the data migration server may define that only unidirectional data migration is allowed. It can be understood that the security and reliability of data can be improved by setting unidirectional data migration.

In an embodiment, a general intermediate table is set in advance for each service data, and the integration platform can migrate each service data conforming to the intermediate table structure to the corresponding general intermediate table.

As step 2 "transfer" in FIG. 4, the integration platform may migrate data conforming to the intermediate table structure into the common intermediate table. The "personal basic information", "address", and "work history" are different types of service data, so each type of service data can be migrated to a common intermediate table, for example, specific personal basic information can be migrated to a common intermediate table set corresponding to the service data such as the "personal basic information". Thereby obtaining various general intermediate tables such as "personal basic information intermediate table", "address intermediate table", "work history intermediate table" and the like in fig. 4.

And step 306, performing adaptation processing corresponding to the target system on the data in the universal intermediate table, and writing the data after the adaptation processing into a target data table in the target system.

The target system is a system to which data is migrated. The adaptation process is a process for adapting data in the general intermediate table to the target system. The target data table is a data table in the target system to which data is migrated.

In one embodiment, the adaptation process may include at least one of a base check, an interface check, and a data conversion process. The basic check is performed on at least one of uniqueness, deletion, and matching with the target system of the data in the general intermediate table. And interface checking, namely checking whether the data in the universal intermediate table accords with the service logic of the interface. And data conversion is conversion processing for adapting the data in the universal intermediate table to the basic data value in the target system.

It will be appreciated that the adaptation process is a data-level transformation of the data in the generic intermediate table, not a data-structural transformation. The data structure mapping transformation process is a transformation on the data structure.

As shown in step 3 "check" in fig. 4, the data in the intermediate table is used to perform adaptation processing corresponding to the target system.

In one embodiment, the data migration server may perform adaptation processing corresponding to the target system on the data in the general intermediate table, and write the data after the adaptation processing into the target data table in the target system. In one embodiment, the data migration server may call a preset interface of the target system, and write the adapted data into a target data table in the target system.

In another embodiment, the target system may have a data migration tool integrated therein, and the common intermediate table may be provided in the data migration tool. The data migration server can perform adaptation processing corresponding to the target system on the data in the universal intermediate table, and write the data after the adaptation processing into the target data table in the target system through the data migration tool. In an embodiment, the data migration server may call a preset interface of the target system through the data migration tool, and write the adapted data into a target data table in the target system.

The preset interface is an interface provided by the target system and used for writing data into the target data table.

In an embodiment, the data migration tool may obtain a target data table identifier returned by the preset interface after calling the preset interface to write the data into the target data table, and perform associated storage on the general intermediate table and the corresponding target data table identifier to establish a corresponding relationship between the general intermediate table and the target data table.

And the target data table identifier is used for uniquely identifying the target data table.

FIG. 5 shows the interaction between the generic intermediate table and the target data table. And the universal intermediate table calls a preset interface to write the data into the target data table, and the target data table returns the target data table identifier to the universal intermediate table.

It is understood that the data in one common intermediate table can be written into a plurality of target data tables in the target system, such as: the data in the address intermediate table can be written into target data tables such as a main address table, a history address table and a history trace address table respectively. Thus, the common intermediate table may correspond to multiple target data table identifications, or to the identification of one core target data table. The core data table is the most core target data table in a plurality of target data tables corresponding to the common intermediate table.

Step 4 "write in" in fig. 4 is a processing step of writing the data after the adaptation processing into a target data table in the target system, namely, calling a preset interface of the target system. In fig. 4, "data import" indicates writing data from the general intermediate table to the target data table, and "result return" indicates returning the target data identification to the general intermediate table.

In one embodiment, the data in the general intermediate table and the data in the corresponding target data table may be verified and compared according to the target data table identifier, and if the comparison is passed, it is determined that the data is actually written. The credibility of the data migration result is improved.

In one embodiment, in an abnormal situation, the data migration process is interrupted, and the data migration may be resumed automatically or manually. In one embodiment, if the data migration server is unexpectedly down, the data migration server may automatically resume performing data migration after restarting by establishing a heartbeat connection. In one embodiment, if the data migration task is always executed due to an abnormality of the network or the database, the user may check whether there is a running data migration thread through the monitor, and after there is no running data migration thread, the data migration tool may copy a new data migration task and execute data migration again.

In the data migration method, according to a preset mapping scheme corresponding to an original system, data structure mapping conversion processing is performed on data to be migrated in the original system to obtain data conforming to an intermediate table structure, the intermediate table structure is a preset data structure required by a universal intermediate table for uniformly interfacing different external systems, therefore, the data of different original systems can be uniformly mapped into the data conforming to the intermediate table structure without writing complex scripts by a user, then the data conforming to the intermediate table structure is migrated into the universal intermediate table, the data in the universal intermediate table is subjected to adaptation processing corresponding to the target system, the data after the adaptation processing is written into a target data table in the target system, therefore, efficient data migration between the original system and the target system with different data structures is realized, and the data migration efficiency is improved.

In one embodiment, before performing data structure mapping conversion on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to the intermediate table structure, the method further includes: determining configured extension migration field entries; an extended migration field entry that is extended and is used to indicate a field entry for data to be migrated; expanding the field items in the universal intermediate table according to the expanded migration field items; acquiring corresponding data in the original system according to the preset migration field item and the extended migration field item to obtain data to be migrated; the preset migration field item is a preset field item used for indicating data to be migrated.

Specifically, a general preset migration field item is preset, a user can configure through a terminal to additionally extend the field item on the basis of the preset migration field item, the data migration server can extend the field item in the general intermediate table according to the extension migration field item configured by the user, and then corresponding data in an original system is acquired according to the preset migration field item and the extension migration field item to obtain data to be migrated.

In one embodiment, a user can configure the extension migration field item through a system which is displayed by a terminal and constructed based on a low-code platform. The configuration can be carried out by a user through a low-code platform only through simple dragging and pulling modes and the like, the operation is convenient and fast, and the expandability of the system is improved.

In the embodiment, on the basis of the preset general preset migration field item, the user can expand the migration field item in a simple configuration mode according to the actual needs of the user, and does not need to write a complex script for expansion, so that the expandability of the system is improved, the convenience of user operation is improved, and the data migration efficiency is improved. In addition, compared with the traditional mode that the personalized content requirement of the user cannot be met by the mode that the spreadsheet is required to be sorted and imported into the target system, the user can expand the migration field, and the flexibility is improved.

In one embodiment, the method further comprises: adding a mark field for the data after the adaptation processing; writing the adapted data into a target data table in a target system comprises: and writing the data which is subjected to the adaptation processing and carries the mark field into a target data table in the target system.

The flag field is a field for recording flag information of data in the general intermediate table. The tag field contains a tag field entry and a tag field value. The flag information is information related to processing of data.

In one embodiment, the tag field may be multiple, and each tag field may record a type of tag information.

In one embodiment, the marking information may include at least one of a completion status of the adaptation process corresponding to the data, a failure reason of the adaptation process, a data update time, a data updater, a data creator, a data creation time, a data batch number, and the like.

In the above embodiment, the data which is subjected to the adaptation processing and carries the mark field is written into the target data table in the target system, so that the user can trace the migration condition of each piece of data according to the mark information in the mark field, thereby improving the security and reliability of the data.

In one embodiment, the adapting the data in the universal intermediate table corresponding to the target system comprises: performing basic check on data in the universal intermediate table; and converting the data in the general intermediate table after the basic verification into target data matched with the basic data value in the target system.

Wherein the basic data value is a preset field value in the target system.

In one embodiment, the adapted data includes target data adapted to a base value in the target system.

In one embodiment, the base check may include at least one of a uniqueness check, a field value absence check, a match check, and the like.

In one embodiment, the data migration tool may translate the data mapping in the base checked universal intermediate table into an identification of the corresponding base material value in the target system.

For example: the field value under the field item of 'professional' in the universal intermediate table is 'algorithm engineer', the basic data value in the target system comprises the field value of 'algorithm engineer', and the corresponding mark of 'algorithm engineer' in the target system is '001', so that the mapping of 'algorithm engineer' in the universal intermediate table can be converted into '001'.

In the embodiment, the data in the universal intermediate table is subjected to basic verification, so that the safety and reliability of the data are improved, and the data in the universal intermediate table after the basic verification are converted into the target data matched with the basic data value in the target system, so that the field value in the universal intermediate table is matched with the basic data value in the target system, and the data in the universal intermediate table can be conveniently written into the target system.

In one embodiment, the base checking of the data in the universal intermediate table comprises at least one of: carrying out uniqueness check on the data in the universal intermediate table; carrying out field value missing verification on mandatory field items in the universal intermediate table; and matching and checking the field values in the universal intermediate table and the corresponding basic data values in the target system.

The mandatory field item is a field item required by the target system to have a field value filled in.

In one embodiment, the universal intermediate table of each business entity has a unique primary key, and the data migration tool can check the uniqueness of the data in the universal intermediate table according to the primary key of the universal intermediate table. In one embodiment, if the verification result is data uniqueness, the uniqueness check passes. And if the verification result is that the data is not unique, feeding back a prompt message for correction.

In one embodiment, the data migration tool may check whether field values of mandatory field entries in the common intermediate table are missing. In one embodiment, if the result of the check is not missing, the field value missing check passes. And if the verification result is missing, feeding back a prompt message to prompt field value supplement.

In one embodiment, the data migration tool may match and check the field values in the common intermediate table with corresponding underlying data values in the target system. And if the matching check result is that the matching is not passed, feeding back a prompt message to prompt that the field value needs to be repaired. For example: the field value under the field item "professional" in the universal intermediate table is "algorithm engineer 1", and the corresponding field value in the target system is "algorithm engineer", the matching is not passed.

In the above embodiment, the uniqueness check is performed on the data in the common intermediate table, so that it can be ensured that the data written in the target system is not duplicated. And performing field value missing check on the mandatory field items in the universal intermediate table, thereby avoiding the missing field values of the mandatory field items and ensuring the integrity and reliability of data. The field value in the universal intermediate table is matched and checked with the corresponding basic data value in the target system, so that the problem that the field value in the intermediate table cannot be mapped and converted into the basic data value in the target system is solved, and the reliability of data migration is improved.

In one embodiment, the structure mapping transformation process is performed by an integrated platform; preset mapping schemes aiming at different external systems are integrated in the integrated platform; the method further comprises the following steps: determining a preset mapping scheme corresponding to an original system from preset mapping schemes integrated in an integration platform and aiming at different external systems; migrating data conforming to the intermediate table structure into the universal intermediate table includes: and migrating the data conforming to the intermediate table structure into the universal intermediate table through the integration platform.

Specifically, the data migration server may determine a preset mapping scheme corresponding to an original system from preset mapping schemes integrated in the integration platform for different external systems, perform data structure mapping conversion processing on data to be migrated in the original system through the integration platform according to the preset mapping scheme corresponding to the original system to obtain data conforming to the intermediate table structure, and migrate the data conforming to the intermediate table structure to the general intermediate table through the integration platform.

In one embodiment, a common intermediate table may be provided in the data migration tool.

In one embodiment, the data migration tool may be integrated into the target system.

In the above embodiment, the integration platform is integrated with preset mapping schemes for different external systems, the preset mapping scheme corresponding to the original system is determined through the integration platform, data structure mapping conversion processing is performed on the data to be migrated in the original system according to the preset mapping scheme corresponding to the original system to obtain data conforming to the intermediate table structure, and then the data conforming to the intermediate table structure is migrated to the general intermediate table, so that the data of different original systems can be uniformly mapped into data conforming to the intermediate table structure without writing a complicated script by a user, efficient data migration between the original system and the target system with different data structures is realized, and the data migration efficiency is improved.

In one embodiment, performing adaptation processing corresponding to the target system on the data in the general intermediate table, and writing the data after the adaptation processing into the target data table in the target system includes: performing adaptation processing corresponding to the target system on the data in the universal intermediate table through a data migration tool integrated in the target system; and calling a preset interface in the target system through the data migration tool, and writing the data after the adaptation processing into a target data table in the target system.

In one embodiment, a common intermediate table may be provided in the data migration tool.

In one embodiment, the integration platform may migrate data conforming to the intermediate table structure into a common intermediate table in the data migration tool.

In the above embodiment, the data migration tool calls the preset interface in the target system, writes the adapted data into the target data table in the target system, and implements data migration in the system layer by calling the preset interface, that is, writes the data in the general intermediate table into multiple target data tables in the target system, rather than performing simple data conversion and data migration between two data tables.

In one embodiment, the method further comprises: determining the dependency relationship among the data in the universal intermediate table according to the code implementation class corresponding to the service type of the data in the universal intermediate table; and according to the dependency relationship, performing adaptation processing corresponding to the target system on the data in the universal intermediate table according to the sequence from the parent-level data to the child-level data, and writing the data after the adaptation processing into the target data table in the target system.

The code implementation class is a specific implementation class for inheriting the service template class. The service template class is a template class which carries out characterization transformation according to the characteristics of different service types on the basis of inheriting the general template class. And the general template class is used for setting the processing steps aiming at the data in the general intermediate table. Dependency refers to the requirement that some data be processed before other data, i.e., the parent data needs to be processed before the child data.

Specifically, the general template class is provided with basic processing steps of a data migration tool, and the basic processing steps comprise: and performing adaptation processing corresponding to the target system on the data in the universal intermediate table, and writing the data after the adaptation processing into a target data table in the target system. And in the service template class, according to the dependency relationship among data in different service types, the characteristic transformation is carried out on the basis of integrating the universal template class.

In one embodiment, in the case of no dependency relationship, the data migration tool may perform, in parallel or in series, a step of performing, on the data in the general intermediate table, adaptation processing corresponding to the target system on the data in the general intermediate table, and writing the data after the adaptation processing into the target data table in the target system, where a user may set, by himself, whether the data is parallel or serial, or set the parallel quantity.

In an embodiment, in a case that a dependency relationship exists between data, the data migration tool may preferentially perform, on parent-level data, a step of performing adaptation processing corresponding to the target system on the data in the general intermediate table, and writing the data after the adaptation processing into a target data table in the target system, and then perform, on child-level data, a step of performing adaptation processing corresponding to the target system on the data in the general intermediate table, and writing the data after the adaptation processing into a target data table in the target system.

For example: for the data of the organization type, it is necessary to process the data of the upper stage first and then the data of the lower stage. And for data of a personnel class, no dependency relationship exists.

As shown in fig. 6, the general template class is provided with basic processing steps of a data migration tool, including the steps of consuming scheduling task messages, writing back task execution states, defining a flow, acquiring data, verifying, converting, calling an interface, verifying data, and the like. The organization template class, the personnel template class, the labor contract template class and the like are service template classes, and the service template classes are characterized and transformed on the basis of integrating general template classes according to the dependency relationship among data in different service types (namely the service types of the same organization, personnel and labor contract). Entity organization migration, post migration, personnel migration, organization personnel migration, labor contract migration and the like, and realizes the code obtained by inheriting the service template class. After receiving the message sent by the message middleware, the data migration tool can perform the steps of data acquisition, verification, conversion, interface calling, data verification and the like on the data in the universal intermediate table according to the code implementation class. The data migration tool can realize the data dependency relationship set in the class by the respective corresponding codes of the data of different service types, control the processing sequence of the data in the universal intermediate table, adopt a serial or parallel mode and control the thread number.

In the embodiment, the reusability is improved by setting the uniform and abstract template class, so that the data migration efficiency is improved. In addition, a distributed cache and a multithreading technology are adopted, optimization is carried out on the process according to the dependency relationship among different data, a front task is completed, a rear task can be started, and the migration tasks of the data without the dependency relationship can be started at the same time and executed in parallel, so that the flexibility and the data migration efficiency are improved.

It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a data migration apparatus for implementing the above-mentioned data migration method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the data migration device provided below may refer to the limitations on the data migration method in the foregoing, and details are not described here.

In one embodiment, as shown in fig. 7, there is provided a data migration apparatus 700, comprising: a data mapping module 702, a data transmission module 704, and a data writing module 706, wherein:

the data mapping module 702 is configured to perform data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system, so as to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly interfacing different external systems.

And a data transmission module 704 for migrating the data conforming to the intermediate table structure into the universal intermediate table.

And a data writing module 706, configured to perform adaptation processing corresponding to the target system on the data in the general intermediate table, and write the data after the adaptation processing into a target data table in the target system.

In one embodiment, as shown in FIG. 8, the data migration apparatus 700 further comprises:

a data acquisition module 708 configured to determine configured extended migration field entries; an extended migration field entry that is extended and is used to indicate a field entry for data to be migrated; expanding the field items in the universal intermediate table according to the expanded migration field items; acquiring corresponding data in the original system according to the preset migration field item and the extended migration field item to obtain data to be migrated; the preset migration field item is a preset field item used for indicating data to be migrated.

In one embodiment, the data writing module 706 is further configured to add a tag field to the data after the adaptation process; and writing the data which is subjected to the adaptation processing and carries the mark field into a target data table in the target system.

In one embodiment, the adapted data includes target data adapted to a base value in the target system; the data writing module 706 is further configured to perform basic check on the data in the general intermediate table; and converting the data in the general intermediate table after the basic verification into target data matched with the basic data value in the target system.

In one embodiment, the data writing module 706 is further configured to perform at least one of the following: carrying out uniqueness check on the data in the universal intermediate table; carrying out field value deletion verification on mandatory field items in the universal intermediate table; and matching and checking the field values in the universal intermediate table and the corresponding basic data values in the target system.

In one embodiment, the structure mapping transformation process is performed by an integrated platform; preset mapping schemes aiming at different external systems are integrated in the integrated platform; the data mapping module 702 is further configured to determine a preset mapping scheme corresponding to the original system from preset mapping schemes integrated in the integration platform and for different external systems; the data transfer module 704 is further configured to migrate data conforming to the intermediate table structure into the common intermediate table via the integration platform.

In one embodiment, the data writing module 706 is further configured to perform adaptation processing corresponding to the target system on the data in the universal intermediate table through a data migration tool integrated in the target system; and calling a preset interface in the target system through the data migration tool, and writing the data after the adaptation processing into a target data table in the target system.

According to the data migration device, according to a preset mapping scheme corresponding to an original system, data structure mapping conversion processing is carried out on data to be migrated in the original system, data conforming to an intermediate table structure is obtained, the intermediate table structure is a preset data structure required by a universal intermediate table for uniformly connecting different external systems, so that the data of different original systems can be uniformly mapped into the data conforming to the intermediate table structure without a user writing a complex script, then the data conforming to the intermediate table structure is migrated into the universal intermediate table, adaptation processing corresponding to a target system is carried out on the data in the universal intermediate table, and the data after the adaptation processing is written into a target data table in the target system, so that efficient data migration between the original system and the target system with different data structures is achieved, and the data migration efficiency is improved.

The modules in the data migration apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data migration method.

Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the above-described method embodiments when executing the computer program.

In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (10)

1. A method of data migration, the method comprising:

performing data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to a middle table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly connecting different external systems;

migrating the data conforming to the intermediate table structure to the universal intermediate table;

and performing adaptation processing corresponding to a target system on the data in the universal intermediate table, and writing the data after the adaptation processing into a target data table in the target system.

2. The method according to claim 1, wherein before performing data structure mapping transformation on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure, the method further comprises:

determining configured extension migration field entries; the extended migration field item is an extended field item used for indicating data to be migrated;

expanding the field items in the universal intermediate table according to the expanded migration field items;

acquiring corresponding data in the original system according to a preset migration field item and the extended migration field item to obtain the data to be migrated; the preset migration field item is a preset field item used for indicating data to be migrated.

3. The method of claim 1, further comprising:

adding a mark field for the data after the adaptation processing;

the writing the data after the adaptation processing into the target data table in the target system includes:

and writing the data which is subjected to the adaptation processing and carries the mark field into a target data table in the target system.

4. The method according to claim 1, wherein the adapting the data in the generic intermediate table corresponding to the target system comprises:

performing basic check on the data in the universal intermediate table;

and converting the data in the universal intermediate table after the basic verification into target data matched with the basic data value in the target system.

5. The method of claim 4, wherein the basic checking of the data in the universal intermediate table comprises at least one of:

carrying out uniqueness check on the data in the universal intermediate table;

carrying out field value deletion verification on mandatory field items in the universal intermediate table;

and matching and checking the field values in the universal intermediate table and the corresponding basic data values in the target system.

6. The method according to any one of claims 1 to 5, wherein the structure mapping conversion process is performed by an integrated platform; preset mapping schemes aiming at different external systems are integrated in the integrated platform; the method further comprises the following steps:

determining a preset mapping scheme corresponding to an original system from preset mapping schemes integrated in an integration platform and aiming at different external systems;

the migrating the data conforming to the intermediate table structure to the universal intermediate table comprises: migrating the data conforming to the intermediate table structure into the universal intermediate table through the integration platform.

7. The method according to any one of claims 1 to 5, wherein the performing adaptation processing corresponding to a target system on the data in the general intermediate table and writing the data after the adaptation processing into a target data table in the target system comprises:

performing adaptation processing corresponding to the target system on the data in the universal intermediate table through a data migration tool integrated in the target system; and calling a preset interface in the target system through the data migration tool, and writing the data after the adaptation processing into a target data table in the target system.

8. An apparatus for data migration, the apparatus comprising:

the data mapping module is used for carrying out data structure mapping conversion processing on the data to be migrated in the original system according to a preset mapping scheme corresponding to the original system to obtain data conforming to an intermediate table structure; the intermediate table structure is a data structure required by a preset universal intermediate table; the universal intermediate table is used for uniformly connecting different external systems;

the data transmission module is used for migrating the data conforming to the intermediate table structure to the universal intermediate table;

and the data writing module is used for performing adaptation processing corresponding to the target system on the data in the universal intermediate table and writing the data after the adaptation processing into a target data table in the target system.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Images