CN114356884A - Data migration method and device - Google Patents

Abstract

The invention discloses a data migration method and device, and relates to the technical field of data storage. One embodiment of the method comprises: determining source data to be migrated in a source system; converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping relationship represents a mapping relationship between fields of the source system and fields of a target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system; and converting the intermediate field into target data with the target system field, and writing the target data into the target system. This embodiment enables a generic data migration scheme.

Description

Data migration method and device

Technical Field

The present invention relates to the field of data storage technologies, and in particular, to a data migration method and apparatus.

Background

After a business system runs for many years, with the continuous increase of business data, the original system architecture can not support business requirements any more, the alternation of a new system is imperative, and the precondition of successful running of the new system is data migration. Data migration refers to migrating stock data of a source system (i.e., an old system) to a target system (i.e., a new system), so that specific business before the source system goes offline continues to be developed at the target system.

The existing data migration scheme mainly comprises the following steps: firstly, determining a migration service range, and analyzing service difference between a source system and a target system; and then determining the business relation of each field of the database table, and submitting the business relation to IT technical personnel to realize a specific migration rule, wherein the whole program is a nonstandard customized program.

In the process of implementing the invention, the inventor finds that the prior art has at least the following problems: firstly, because the source system of each data migration is different, each data migration is a specific customized migration scheme, and a general solution cannot be formed, so that the developed migration tool is generally a one-time tool, thereby causing a large amount of repeated development and increasing the workload of technicians. Secondly, the existing data migration scheme needs the close cooperation of service personnel and technical personnel to realize each rule by using codes after the rule is clear, and finally the realization of the migration rule is finished.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data migration method and apparatus, which can implement a general data migration scheme.

To achieve the above object, according to one aspect of the present invention, a data migration method is provided.

The data migration method is used for migrating the source data in at least one source system to a target system; the method comprises the following steps: determining source data to be migrated in the source system; converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping represents a mapping between fields of the source system and fields of the target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system; and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

Optionally, the field mapping relationship is configured based on preset configuration templates, and each configuration template corresponds to one field mapping form.

Optionally, the field mapping relationship includes: a field name mapping relationship and a field value mapping relationship between the source system field and the target system field.

Optionally, the fields of the target system further comprise non-essential fields, the intermediate data having partial non-essential fields of the target system; and, the method further comprises: first performing data migration for the required fields of the target system; after the data migration of the required fields is completed, data migration is performed for the non-required fields.

Optionally, the determining source data to be migrated in the source system includes: determining data of the source system in a preset time range as initial selection data; and identifying and removing abnormal data in the primary selection data to form the source data to be migrated.

Optionally, the field mapping form includes: the mapping between fields of multiple source systems and fields of a target system, the mapping between fields of multiple tables of the same source system and fields of a target system, the mapping between fields of the same table of the source system and fields of a target system, and the mapping between a field of the same table of the source system and fields of a target system.

Optionally, the number of the source systems is multiple, the field mapping relationship includes a first field mapping relationship, and the first field mapping relationship represents a mapping relationship between fields of multiple source systems and fields of a target system; the intermediate data comprises first data having a target system field in a first field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including: and performing conversion of the source data across a plurality of source systems according to the first field mapping relation to form first data.

Optionally, the field mapping relationship includes a second field mapping relationship, where the second field mapping relationship represents a mapping relationship between fields in multiple tables of the same source system and fields of the target system; the intermediate data comprises second data having a target system field in a second field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including: and performing conversion of multiple tables in the same source system on the source data according to the second field mapping relation, and combining the first data to form second data.

Optionally, the field mapping relationship includes a third field mapping relationship, where the third field mapping relationship represents a mapping relationship between a plurality of fields in the same table of the source system and a field of the target system; the intermediate data comprises third data having a target system field in a third field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including: and performing conversion of multiple fields in the same table of the source system on the source data according to a third field mapping relation, and combining second data to form third data.

Optionally, the field mapping relationship includes a fourth field mapping relationship, where the fourth field mapping relationship represents a mapping relationship between a field in the same table of the source system and a field of the target system; the intermediate data comprises fourth data having a target system field in a fourth field mapping; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including: and performing conversion on the source data according to a fourth field mapping relation, and combining the third data to form fourth data.

Optionally, the non-required fields of the target system include a new field that does not correspond to the field of the source system, and the intermediate data does not have the new field; and, said converting said intermediate field into target data having said target system field comprises: adding the data of the newly added field in fourth data to form the target data; and generating the value of the target data in the field of the newly added field according to the preset logic of the target system.

To achieve the above object, according to another aspect of the present invention, there is provided a data migration apparatus.

The data migration device of the embodiment of the invention is used for migrating the source data in at least one source system to a target system; the apparatus may include: a data acquisition unit for: determining source data to be migrated in the source system; a conversion unit to: converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping represents a mapping between fields of the source system and fields of the target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system; a data writing unit to: and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

To achieve the above object, according to still another aspect of the present invention, there is provided an electronic apparatus.

An electronic device of the present invention includes: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the data migration method provided by the invention.

To achieve the above object, according to still another aspect of the present invention, there is provided a computer-readable storage medium.

A computer-readable storage medium of the present invention has stored thereon a computer program which, when executed by a processor, implements the data migration method provided by the present invention.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer program product.

A computer program product according to an embodiment of the present invention includes a computer program, and when the computer program is executed by a processor, the computer program implements the data migration method according to an embodiment of the present invention.

According to the technical scheme of the invention, the embodiment of the invention has the following advantages or beneficial effects:

firstly, determining source data to be migrated in a source system; then, converting the source data into at least one piece of intermediate data according to a preset field mapping relation; the field mapping relation is configured according to a preset configuration template, and each configuration template corresponds to a field mapping form; thereafter, the intermediate field is converted into target data having a target system field, and the target data is written to the target system. Through the steps, a universal data migration scheme can be realized by using the configuration template, after the development of technical personnel is completed, business personnel only need to select a proper configuration template to write a specific field mapping relation and load the field mapping relation, and data migration can be independently executed according to business requirements, so that repeated development of a data migration program is avoided. In addition, because the conversion complexity difference between different source data and target system data is large, multiple conversion processes can be set based on intermediate data, the same conversion process can execute the same data processing, for example, a certain conversion process executes field mapping across a source system, another conversion process executes field mapping across a database table and the same source database, and a conversion process executes multi-field mapping of the same database table, thereby improving data migration reliability and avoiding data errors.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of the main steps of a data migration method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an execution architecture of a data migration method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a portion of a data migration apparatus according to an embodiment of the present invention;

FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 5 is a schematic structural diagram of an electronic device for implementing the data migration method in the embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

It should be noted that the embodiments of the present invention and the technical features of the embodiments may be combined with each other without conflict.

FIG. 1 is a schematic diagram of the main steps of a data migration method according to an embodiment of the present invention.

As shown in fig. 1, the data migration method according to the embodiment of the present invention may specifically be executed according to the following steps:

step S101: source data to be migrated in a source system is determined.

In the embodiment of the present invention, source data in at least one source system needs to be migrated to a target system, one source system may have at least one database, and one target system may also have at least one database, and the following description will take an example in which each source system and each target system has one database respectively.

In specific application, a corresponding Structured Query Language (SQL) statement can be written and executed in a source system, so that source data to be migrated can be queried. Considering that more remote historical data may have less correlation with the current service, data of the source system in a preset time range (for example, the last year) may be determined as primary data, and abnormal data in the primary data may be identified and removed, so as to form source data to be migrated. Illustratively, the abnormal data in the initially selected data can be automatically and quickly identified by using a pre-trained abnormal data detection model.

Step S102: and converting the source data into at least one piece of intermediate data according to the preset field mapping relation.

In this step, the intermediate data is data of an intermediate state from the source data to the target data (i.e., data in the target system), and the intermediate data may have all necessary fields of the target system (i.e., fields necessary for the target system to operate, and accordingly, the target system may also have unnecessary fields), or may have partial unnecessary fields of the target data. The conversion of one or more intermediate data is set between the source data and the target data, so as to avoid data errors caused by directly executing a relatively complex conversion logic between the source data and the target data, and only the same type of data processing can be executed in the same conversion process by setting a plurality of intermediate data and corresponding multiple conversion processes, for example, a certain conversion process executes field mapping across a source system, another conversion process executes field mapping across a source database and a database table, and a conversion process executes multi-field mapping of the same database table, thereby being beneficial to ensuring the reliability and stability of the data conversion process and avoiding generating data errors.

In practical applications, each data conversion needs to rely on the above field mapping relationship, which may be embodied in data of the configuration file, and represents a mapping relationship between a field of the source system and a field of the target system, where the mapping relationship may include a field name mapping relationship and a field value mapping relationship. For example, mapping the field name stu of the source system to the field name stu1 of the target system belongs to the field name mapping relationship. The field value mapping relationship comprises a format mapping relationship and a numerical value mapping relationship, wherein the format mapping relationship is related to the format requirement of each system on numerical values, for example, three digits in a source system need to be mapped into nine digits of a target system, and if 001 needs to be mapped into 010000000; the CNY in the source system needs to be mapped 156 to the target system. The value mapping relationship is generally found in the case of one-to-many (i.e. one field of the source system corresponds to multiple fields of the target system), many-to-one (i.e. multiple fields of the source system correspond to one field of the target system), and many-to-many (i.e. multiple fields of the source system correspond to multiple fields of the target system), and is used for calculating the values of the relevant fields, for example, a value mapping system is: for the corresponding record, the value of the amount actually received by the source system minus the value of the amortized amount field divided by 100 equals the value of the amount to be amortized field of the target system.

Preferably, the field mapping relationship may be configured based on a preset configuration template, that is, a configuration file embodying the field mapping relationship is generated by using the configuration template. The configuration template can be developed by technical personnel, and business personnel can use the configuration template to realize specific data migration rules by combining with actual business requirements, so that data migration work is independently completed. In an actual scenario, each configuration template may correspond to one field mapping form, and the field mapping forms may include the following four types: the mapping between multiple fields of multiple source systems and target system fields (target system fields may be one or more), the mapping between multiple fields of multiple tables of the same source system and target system fields (target system fields may be one or more), the mapping between multiple fields of the same table of the source system and target system fields (target system fields may be one or more), and the mapping between one field of the same table of the source system and target system fields (target system fields may be one or more). When the business personnel use the configuration template, the business personnel can select a proper template according to the field mapping condition and write the template so as to generate a corresponding configuration file.

In an alternative embodiment, the above required fields and non-required fields of the target system have different data migration priorities. It can be understood that in an actual scenario, the source data to be migrated often has hundreds of thousands of fields, and it takes a long time for the full migration, in a case of a time emergency, the data of the necessary fields may be migrated first to ensure that the target system can operate as soon as possible, and the data of the unnecessary fields may be migrated gradually after the data migration of the necessary fields is completed.

Step S103: and converting the intermediate field into target data with a target system field, and writing the target data into the target system.

In the embodiment of the present invention, the unnecessary fields of the target system may include a new field that does not correspond to any field of the source system, and the above intermediate data does not have a new field, for example, an update time field of the target system is the new field, and the field value needs to be determined according to the time corresponding to the latest operation. In this step, data of the newly added field may be added to the last intermediate data, thereby forming target data. It is understood that the above field values of the target data in the added fields may be generated according to preset logic of the target system (e.g., the above logic of generating the update time).

One embodiment of the present invention is explained below.

In this embodiment, there is only one piece of intermediate data in the process from the source data to the target data, and the intermediate data has all necessary fields and some unnecessary fields of the target system. The fields of the source data are as follows:

organization number

Coin kind

Total amount of contract

Contract start date

Amortized amount of money

Actual amount of money

The fields of the target data are as follows:

wherein, the organization identification (corresponding to the organization number of the source data), currency (corresponding to the currency of the source data), contract amount (corresponding to the total contract amount of the source data), and amount to be amortized (corresponding to the amortized amount and the real amount of the source data) are required fields, the contract start date (corresponding to the contract start date of the source data) is a non-required field, and the contract number, the teller number, and the update time are newly added fields (not corresponding to any field of the source data).

The following intermediate data fields may be set:

mechanism identification

Coin kind

Amount of contract

Contract start date

Amount to be shared

And selecting a corresponding configuration template to configure the field name mapping relationship, wherein the format mapping relationship in the field name mapping relationship can be as follows: the value of the source data authority number maps to the value of the target data authority identification, e.g., 001 maps to 010000000; the value of the source data currency maps to the value of the target data currency, e.g., CNY maps to 156; the value of the source data contract total is divided by 100 to obtain the value of the target data contract amount (the amount units before and after mapping are different).

The numerical mapping in the field value mapping may be as follows: the value of the real amount of the source data minus the value of the amortized amount divided by 100 equals the value of the amount to be amortized for the target data.

After the field mapping relation is established, the source data to be migrated in the source system can be inquired, and the field mapping relation is automatically converted into intermediate data. After the intermediate data is obtained, three fields of contract number, teller number and update time can be added to form target data (the above process can be realized by using a program and does not depend on the field mapping relation). The values of the contract number and teller number fields can be determined according to preset logic of the target system, and if it cannot be determined that the corresponding data does not exist in the source system at the same time, the two fields can also be null values. The value of the update time field is equal to the machine time when the operation was performed. And finally, writing the target data into the target system, namely realizing data migration.

It can be seen that, for different data migration tasks, the change of the conversion logic from the source data to the intermediate data is large, and the change of the conversion logic from the intermediate data to the target data is small, so that the logic with large change and the logic with small change can be separated, the logic with large change is realized through codes (configuration files can not be written), and the logic with small change is expressed by using a field mapping relation, which is helpful for avoiding repeated development of the same content and forming a generalized data migration scheme.

Another embodiment of the present invention is described below, as shown in fig. 2.

In the present embodiment, there are four pieces of intermediate data in the process from the source data to the target data, which are the first data, the second data, the third data, and the fourth data, respectively. The conversion from source data to first data uses a first field mapping relationship (it is understood that the first data has a target system field in the first field mapping relationship), the conversion from source data to second data uses a second field mapping relationship (it is understood that the second data has a target system field in the second field mapping relationship), the conversion from source data to third data uses a third field mapping relationship (it is understood that the third data has a target system field in the third field mapping relationship), and the conversion from source data to fourth data uses a fourth field mapping relationship (it is understood that the fourth data has a target system field in the fourth field mapping relationship). Wherein each field mapping relationship represents the same type of conversion mode.

For example, a first field mapping involves a cross-source system transformation that represents a mapping between fields of multiple source systems and fields of a target system; the second field mapping relation relates to cross-table conversion and represents the mapping relation between fields in multiple tables of the same source system and fields of the target system; the third field mapping relation relates to cross-field conversion and represents the mapping relation between a plurality of fields in the same table of the source system and the fields of the target system; the fourth field mapping is a single field conversion of the source system, which represents the mapping between a field in the same table of the source system and the field of the target system.

When data migration is executed, conversion spanning multiple source systems can be executed on source data according to a first field mapping relation to form first data; then, performing conversion of multiple tables in the same source system on the source data according to a second field mapping relation, and adding first data in a conversion result to form second data; then, according to the mapping relation of the third field, the conversion of multiple fields in the same table of the cross-source system is executed on the source data, and second data is added in the conversion result to form third data; then, according to a fourth field mapping relation, carrying out conversion from a single field of the source data to a field of a target system on the source data, and adding third data in a conversion result to form fourth data; and finally, adding data of the newly added field in the fourth data to form target data, and writing the target data into a target database to realize data migration.

In the technical scheme of the embodiment of the invention, the configuration template is a general template irrelevant to the service, and service personnel can use the configuration template to realize specific service requirements.

It should be noted that, for the convenience of description, the foregoing method embodiments are described as a series of acts, but those skilled in the art will appreciate that the present invention is not limited by the order of acts described, and that some steps may in fact be performed in other orders or concurrently. Moreover, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no acts or modules are necessarily required to implement the invention.

To facilitate a better implementation of the above-described aspects of embodiments of the present invention, the following also provides relevant means for implementing the above-described aspects.

Referring to fig. 3, a data migration apparatus 300 according to an embodiment of the present invention is used for migrating source data in at least one source system to a target system, and may include: a data acquisition unit 301, a conversion unit 302, and a data writing unit 303.

Wherein, the data obtaining unit 301 is configured to: determining source data to be migrated in the source system; the conversion unit 302 may be configured to: converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping represents a mapping between fields of the source system and fields of the target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system; the data writing unit 303 may be configured to: and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

In the embodiment of the invention, the field mapping relationship is configured based on preset configuration templates, and each configuration template corresponds to one field mapping form.

As a preferred scheme, the field mapping relationship includes: a field name mapping relationship and a field value mapping relationship between the source system field and the target system field.

Preferably, the fields of the target system further comprise non-essential fields, the intermediate data having partial non-essential fields of the target system; and, the apparatus 300 may further comprise an execution unit to: first performing data migration for the required fields of the target system; after the data migration of the required fields is completed, data migration is performed for the non-required fields.

In a specific application, the data obtaining unit 301 may further be configured to: determining data of the source system in a preset time range as initial selection data; and identifying and removing abnormal data in the primary selection data to form the source data to be migrated.

In practical applications, the field mapping form includes: the mapping between fields of multiple source systems and fields of a target system, the mapping between fields of multiple tables of the same source system and fields of a target system, the mapping between fields of the same table of the source system and fields of a target system, and the mapping between a field of the same table of the source system and fields of a target system.

In one embodiment, the number of the source systems is multiple, the field mapping relationship comprises a first field mapping relationship, and the first field mapping relationship represents a mapping relationship between fields of multiple source systems and fields of a target system; the intermediate data comprises first data having a target system field in a first field mapping relationship; and, the conversion unit 302 may be further configured to: and performing conversion of the source data across a plurality of source systems according to the first field mapping relation to form first data.

In an optional implementation manner, the field mapping relationship includes a second field mapping relationship, and the second field mapping relationship represents a mapping relationship between fields in multiple tables of the same source system and fields of the target system; the intermediate data comprises second data having a target system field in a second field mapping relationship; and, the conversion unit 302 may be further configured to: and performing conversion of multiple tables in the same source system on the source data according to the second field mapping relation, and combining the first data to form second data.

In some embodiments, the field mapping includes a third field mapping representing a mapping between a plurality of fields in the same table of the source system and a field of the target system; the intermediate data comprises third data having a target system field in a third field mapping relationship; and, the conversion unit 302 may be further configured to: and performing conversion of multiple fields in the same table of the source system on the source data according to a third field mapping relation, and combining second data to form third data.

In a specific scenario, the field mapping relationship includes a fourth field mapping relationship, and the fourth field mapping relationship represents a mapping relationship between a field in the same table of the source system and a field of the target system; the intermediate data comprises fourth data having a target system field in a fourth field mapping; and, the conversion unit 302 may be further configured to: and performing conversion on the source data according to a fourth field mapping relation, and combining the third data to form fourth data.

In addition, in the embodiment of the present invention, the non-required fields of the target system include a new field that does not correspond to the field of the source system, and the intermediate data does not have the new field; and, the data writing unit 303 may be further configured to: adding the data of the newly added field in fourth data to form the target data; and generating the value of the target data in the field of the newly added field according to the preset logic of the target system.

In the technical scheme of the embodiment of the invention, a universal data migration scheme is provided, business personnel can independently complete the formulation of a data migration rule by using a configuration template without customizing and compiling a specific migration rule by technical personnel, so that the code reuse rate is improved, and development resources are saved.

Fig. 4 illustrates an exemplary system architecture 400 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.

As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405 (this architecture is merely an example, and the components included in a particular architecture may be adapted according to application specific circumstances). The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables.

A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as a data migration application (for example only).

The terminal devices 401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 405 may be a server providing various services, such as a background server (for example only) providing support for a data migration application operated by a user with a terminal device 401, 402, 403. The backend server may process the received data migration request and the like, and feed back a processing result (e.g., a data migration result, for example only) to the terminal devices 401, 402, and 403.

It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the data migration apparatus is generally disposed in the server 405.

It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The invention also provides the electronic equipment. The electronic device of the embodiment of the invention comprises: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the data migration method provided by the invention.

Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use in implementing an electronic device of an embodiment of the present invention. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the computer system 500 are also stored. The CPU501, ROM 502, and RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.

In particular, the processes described in the main step diagrams above may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the invention include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the main step diagram. In the above-described embodiment, the computer program can be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the central processing unit 501.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a data acquisition unit, a conversion unit, and a data writing unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, the data acquisition unit may also be described as a "unit that provides source data to be migrated to the conversion unit".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to perform steps comprising: determining source data to be migrated in a source system; converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping relationship represents a mapping relationship between fields of the source system and fields of a target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system; and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

A computer program product according to an embodiment of the present invention includes a computer program that, when executed by a processor, implements a data migration method in the practice of the present invention.

In the technical scheme of the embodiment of the invention, a universal data migration scheme can be realized by using the configuration template, and after the development of technical personnel is completed, business personnel can independently execute data migration according to business requirements by only selecting a proper configuration template to write a specific field mapping relation and loading the field mapping relation, thereby avoiding the repeated development of a data migration program. In addition, because the conversion complexity difference between different source data and target system data is large, multiple conversion processes can be set based on intermediate data, the same conversion process can execute the same data processing, for example, a certain conversion process executes field mapping across a source system, another conversion process executes field mapping across a database table and the same source database, and a conversion process executes multi-field mapping of the same database table, thereby improving data migration reliability and avoiding data errors.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (15)

1. A data migration method is used for migrating source data in at least one source system to a target system; characterized in that the method comprises:

determining source data to be migrated in the source system;

converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the content of the first and second substances,

the field mapping relationship represents a mapping relationship between a field of the source system and a field of the target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system;

and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

2. The method of claim 1, wherein the field mapping relationship is configured based on preset configuration templates, and each configuration template corresponds to a field mapping form.

3. The method of claim 2, wherein the field mapping comprises: a field name mapping relationship and a field value mapping relationship between the source system field and the target system field.

4. The method of claim 3, wherein the fields of the target system further comprise non-essential fields, and wherein the intermediate data has partial non-essential fields of the target system; and, the method further comprises:

first performing data migration for the required fields of the target system;

after the data migration of the required fields is completed, data migration is performed for the non-required fields.

5. The method of claim 1, wherein the determining source data to be migrated in the source system comprises:

determining data of the source system in a preset time range as initial selection data;

and identifying and removing abnormal data in the primary selection data to form the source data to be migrated.

6. The method of claim 4, wherein the field mapping form comprises: the mapping between fields of multiple source systems and fields of a target system, the mapping between fields of multiple tables of the same source system and fields of a target system, the mapping between fields of the same table of the source system and fields of a target system, and the mapping between a field of the same table of the source system and fields of a target system.

7. The method of claim 6, wherein the number of the source systems is plural, the field mapping comprises a first field mapping, and the first field mapping represents a mapping between fields of plural source systems and fields of a target system; the intermediate data comprises first data having a target system field in a first field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including:

and performing conversion of the source data across a plurality of source systems according to the first field mapping relation to form first data.

8. The method of claim 7, wherein the field mapping comprises a second field mapping representing a mapping between fields in multiple tables of the same source system and fields of the target system; the intermediate data comprises second data having a target system field in a second field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including:

and performing conversion of multiple tables in the same source system on the source data according to the second field mapping relation, and combining the first data to form second data.

9. The method of claim 8, wherein the field mapping comprises a third field mapping, and the third field mapping represents a mapping between a plurality of fields in the same table of the source system and a field of the target system; the intermediate data comprises third data having a target system field in a third field mapping relationship; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including:

and performing conversion of multiple fields in the same table of the source system on the source data according to a third field mapping relation, and combining second data to form third data.

10. The method of claim 9, wherein the field mapping comprises a fourth field mapping, and the fourth field mapping represents a mapping between a field in the same table of the source system and a field of the target system; the intermediate data comprises fourth data having a target system field in a fourth field mapping; and converting the source data into at least one piece of intermediate data according to a pre-configured field mapping relationship, including:

and performing conversion on the source data according to a fourth field mapping relation, and combining the third data to form fourth data.

11. The method of claim 10, wherein the non-required fields of the target system include an added field that does not correspond to the field of the source system, and wherein the intermediate data does not have the added field; and, said converting said intermediate field into target data having said target system field comprises:

adding the data of the newly added field in fourth data to form the target data; and generating the value of the target data in the field of the newly added field according to the preset logic of the target system.

12. A data migration device is used for migrating source data in at least one source system to a target system; characterized in that the device comprises:

a data acquisition unit for: determining source data to be migrated in the source system;

a conversion unit to: converting the source data into at least one piece of intermediate data according to a preset field mapping relation; wherein the field mapping represents a mapping between fields of the source system and fields of the target system; the fields of the target system include required fields, the intermediate data having the required fields of the target system;

a data writing unit to: and converting the intermediate field into target data with the target system field, and writing the target data into the target system.

13. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-11.

14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-11.

15. A computer program product comprising a computer program, characterized in that the computer program realizes the method according to any of claims 1-11 when executed by a processor.

Images